As health systems continue to deal with workforce shortages, many leaders are turning to technology to try to alleviate some of the burdens on their existing staff. The Guthrie Clinic, based in Sayre, Pa., created a virtual care command center, dubbed the ...

How health systems are using tech to address staff shortages

By Rob Curran IQVIA Holdings posted 7% third-quarter net-income growth as sales of its analytic and research services to laboratories grew. The

IQVIA Net Up on Strong Demand for Life-Sciences Services

IKS Health, a company that provides administrative, clinical, operational and revenue tech solutions, acquired AQuity Solutions for $200 million to broaden its services from ambulatory care providers to hospitals.

Health tech company IKS Health picks up AQuity for $200M to broaden its reach to hospitals

The British Library has said a “major technology outage” has hit online services, public wifi at the site and its website. The research library in London, which has a large deposit of the UK and world’s book collections, is investigating the issue.

‘Major technology outage’ reported at British Library after cyber incident

and GenAI applications using the latest foundation models from Amazon Web Services (AWS), Hugging Face, OpenAI, Meta, and others. Domino and NVIDIA are creating templates based on the NVIDIA NeMo framework and other NVIDIA AI software to help developers ...

Domino Data Lab Unveils New Generative AI Capabilities in Latest Platform Release

Artificial intelligence has been all the rage in almost every industry. This #OverlandPark company hopes to capitalize on that with its new offering. #AI #healthcare

Netsmart's new offering aims to add efficiency, alleviate labor shortages in health care

The Global Process Analytical Technology Market is estimated to add an opportunity of US$ 4,923.41 million from 2022 to 2030. The global market is

Process Analytical Technology Market to Surpass at a Revenue of US$ 7,828.03 Million By 2031 | Astute Analytica

Wariness about being impersonated online and scepticism about the benefits of artificial intelligence is thwarting hopes to use it to offer advanced personalised services.

Fear of fakes fuels deep distrust in AI

In the digital age, where a single online review or social media post can sway public opinion and consumer behavior, managing your online reputation has become critical, necessitating sophisticated tools and strategies.

Exclusive Report: How AI’s Analytical Mastery Help With Online Reputation Management

Although English is not his first language, Whitehouse High School senior Sebastian Toro doesn’t feel excluded from classes or interacting with his peers. And it’s all due to

Whitehouse ISD benefits from technology that provides real-time translation for students

Analytica

analytica

Cambridge Analytica's $1B Mistake: The Psychographics-as-a-Service Market Nobody Dares to Build

Advancing the Detection of Ultralight Vector Dark Matter through Novel Analytical Strategies

Developers can leverage advanced analytics tools to streamline and improve software, compliance, and internal controls auditing. Several novel solutions solve existing pain points, suggesting their effect on audit processes could be permanent. How Advanced Analytics Enhances Audit Processes Whether auditors review contracts, financial statements, code, or internal controls, their traditional techniques are tedious — and inherently inaccurate to some degree. Take sampling, for example, where they test a database subset. It is not uncommon for sampling portions to be trivial compared to the total population. For example, while testing 500,000 Amazon transactions may seem significant, it only accounts for 0.012% of the total — which may introduce unintended bias or errors, skewing audit process insights. Advanced analytics tools like artificial intelligence, complex event processing, and data mining enable 100% population testing because of their rapid processing capabilities. They eliminate the need for sampling, thereby reducing bias and error risks. These tools can also enhance fieldwork, risk assessment, document reviews, and report drafting. Autonomous technologies like AI are particularly beneficial since automation eliminates human error and increases quality. Moreover, they enable real-time data analysis or anomaly detection, streamlining processes. Examples of Successful Analytics-Based Auditing The Internal Revenue Service (IRS) recently leveraged advanced analytics for its audits. Since they are relatively rare, few people are audited yearly. While this undoubtedly benefits individuals, it creates persistent discrepancies and financial losses for the federal government. The U.S. Government Accountability Office reports that the IRS is using AI-powered tools to close the multi-billion dollar tax gap. It uses multiple models to audit select taxpayers and partnerships by selecting large representative samples of high-risk returns. This automation eliminates tedious manual processes, enabling federal employees to expand their scope. Many leading accounting firms have also seen success. For instance, PricewaterhouseCoopers has deployed AI auditing technology for financial procedure review to identify irregularities, enabling it to perform more in-depth analysis. Similarly, Ernst & Young uses natural language processing (NLP) technology to minimize the need for manual contract reviews. It detects anomalies during review and explains its findings, minimizing the risk of material misstatements. A human-in-the-loop validates its model’s output, offering expertise and context awareness to maximize precision. Deloitte developed an AI-powered chatbot named DARTbot for 18,000 audit employees, enabling real-time assistance. It provides source material alongside output, enhancing explainability and enabling auditing teams to analyze complex queries in greater detail. Klynveld Peat Marwick Goerdeler — one of the largest global accounting firms — partnered with Mindbridge, an AI platform, to increase audit explainability. Its advanced rule-based and machine-learning technology integrates with the auditing platform KPMG Clara for real-time analysis. Consequently, its auditors gain more time to review high-risk items. What Technology Can Be Used to Improve Auditing? While many notable case studies revolve around financial, compliance, and consulting applications, this approach is also relevant for software auditing. Developers can leverage cutting-edge analytical tech to improve accuracy, explainability, and speed. 1. Data Visualization With data visualization, developers can convert databases into graphics, charts, and tables, enhancing vendor collaboration and providing clarity to auditees. Since tools like AI can process unstructured data, they are not bound to text-based format. 2. Complex Event Processing Since complex event processing analyzes numerous systems simultaneously in real-time, developers can detect software abnormalities swiftly. Consequently, pinpointing the source of issues becomes much more manageable. 3. Artificial Intelligence AI tools can accelerate most auditing processes because of their autonomous and rapid processing capabilities. Developers can even use generative models or algorithm-powered chatbots to rewrite code or suggest solutions for errors. The Benefits of Leveraging Advanced Analytical Tools Developers using advanced analytical tools benefit from being able to audit in real-time. Moreover, they can comprehensively review entire populations instead of sampling data. These benefits contribute to a more accurate, precise, and quality assessment. Developers can leverage advanced analytics tools to streamline and improve software, compliance, and internal controls auditing. Several novel solutions solve existing pain points, suggesting their effect on audit processes could be permanent. How Advanced Analytics Enhances Audit Processes Whether auditors review contracts, financial statements, code, or internal controls, their traditional techniques are tedious — and inherently inaccurate to some degree. Take sampling, for example, where they test a database subset. It is not uncommon for sampling portions to be trivial compared to the total population. For example, while testing 500,000 Amazon transactions may seem significant, it only accounts for 0.012% of the total — which may introduce unintended bias or errors, skewing audit process insights. accounts for 0.012% of the total accounts for 0.012% of the total Advanced analytics tools like artificial intelligence, complex event processing, and data mining enable 100% population testing because of their rapid processing capabilities. They eliminate the need for sampling, thereby reducing bias and error risks. These tools can also enhance fieldwork, risk assessment, document reviews, and report drafting. Autonomous technologies like AI are particularly beneficial since automation eliminates human error and increases quality. Moreover, they enable real-time data analysis or anomaly detection, streamlining processes. automation eliminates human error automation eliminates human error Examples of Successful Analytics-Based Auditing The Internal Revenue Service (IRS) recently leveraged advanced analytics for its audits. Since they are relatively rare , few people are audited yearly. While this undoubtedly benefits individuals, it creates persistent discrepancies and financial losses for the federal government. they are relatively rare they are relatively rare The U.S. Government Accountability Office reports that the IRS is using AI-powered tools to close the multi-billion dollar tax gap. It uses multiple models to audit select taxpayers and partnerships by selecting large representative samples of high-risk returns. This automation eliminates tedious manual processes, enabling federal employees to expand their scope. is using AI-powered tools is using AI-powered tools Many leading accounting firms have also seen success. For instance, PricewaterhouseCoopers has deployed AI auditing technology for financial procedure review to identify irregularities, enabling it to perform more in-depth analysis. deployed AI auditing technology deployed AI auditing technology Similarly, Ernst & Young uses natural language processing (NLP) technology to minimize the need for manual contract reviews. It detects anomalies during review and explains its findings, minimizing the risk of material misstatements. A human-in-the-loop validates its model’s output , offering expertise and context awareness to maximize precision. validates its model’s output validates its model’s output Deloitte developed an AI-powered chatbot named DARTbot for 18,000 audit employees , enabling real-time assistance. It provides source material alongside output, enhancing explainability and enabling auditing teams to analyze complex queries in greater detail. for 18,000 audit employees for 18,000 audit employees Klynveld Peat Marwick Goerdeler — one of the largest global accounting firms — partnered with Mindbridge, an AI platform, to increase audit explainability. Its advanced rule-based and machine-learning technology integrates with the auditing platform KPMG Clara for real-time analysis. Consequently, its auditors gain more time to review high-risk items. integrates with the auditing platform integrates with the auditing platform What Technology Can Be Used to Improve Auditing? While many notable case studies revolve around financial, compliance, and consulting applications, this approach is also relevant for software auditing. Developers can leverage cutting-edge analytical tech to improve accuracy, explainability, and speed. 1. Data Visualization With data visualization, developers can convert databases into graphics, charts, and tables, enhancing vendor collaboration and providing clarity to auditees. Since tools like AI can process unstructured data, they are not bound to text-based format. 2. Complex Event Processing Since complex event processing analyzes numerous systems simultaneously in real-time, developers can detect software abnormalities swiftly. Consequently, pinpointing the source of issues becomes much more manageable. 3. Artificial Intelligence AI tools can accelerate most auditing processes because of their autonomous and rapid processing capabilities. Developers can even use generative models or algorithm-powered chatbots to rewrite code or suggest solutions for errors. The Benefits of Leveraging Advanced Analytical Tools Developers using advanced analytical tools benefit from being able to audit in real-time. Moreover, they can comprehensively review entire populations instead of sampling data. These benefits contribute to a more accurate, precise, and quality assessment.

Enhancing Audit Processes With Advanced Analytical Tools

Before making any investments, you should ensure to conduct your own thorough research and due diligence. Investors are excited to see where the will develop. This $ SPONGEV2 forecast looks at the trends and variables that could influence the Sponge V2 price. Read on to find out how SPONGEV2 can adapt to the dynamic crypto market. Sponge Token V2 Sponge Token V2 Price Development Forecast Experienced crypto investors are currently closely following the new Sponge Token V2 price. Although it is difficult to predict future value, we will look at some possibilities for SPONGEv2 price development: We assume that the growth trend of Sponge Token V2 will initially increase in 2024. Market experts predict positive momentum for the currency, fueled by increasing demand, practicality, and community support. Although assessments vary, SPONGEV2 has a bright future ahead of it, with the possibility of price appreciation. Nevertheless, investors should proceed with caution when investing and obtain comprehensive information. End of 2024: By 2025, Sponge Token V2 should have reached its full potential in the crypto sector. Its price rise, analysts say, will be heavily influenced by its usefulness, adoption, and community engagement. Any Sponge Token V2 prediction beyond 2025 is somewhat speculative, but the cryptocurrency could continue its rise this year. Stay informed and think seriously about your financial strategy. End of 2025: Sponge Token V2 may have become a major participant in the crypto market by 2030. The $SPONGEV2 coin has the potential to see significant price appreciation if it has a robust community, practical usage, and widespread adoption. The path that $SPONGEV2 takes until 2030 could be promising for long-term investors in Germany, even if exact price predictions are not yet known. The crypto market is constantly changing. Therefore, it is important to be attentive and flexible. End of 2030: By 2030, $SPONGEV2 could have changed a lot and its value could be affected by many things. The price will depend heavily on how widespread it is as a utility token, what relationships it forms with organizations and platforms, and what value it has for consumers. In an ideal situation, SPONGEV2 would see massive adoption and usage and eventually become an essential part of the blockchain ecosystem. For this reason, the price could rise to unprecedented heights. However, since the crypto market is often volatile, you have to be careful. It is important for long-term investors to follow SPONGEV2's development path, collaborations, and community support. To successfully manage risk in the current crypto market, diversification is still a good approach. Before adding SPONGEV2 to your portfolio, you should seek financial advice, just as you would with any other investment. Possible Price Developments for Sponge Token V2 Many factors influence the crypto market, which in turn affects the price of the Sponge Token V2. Although it is difficult to determine the exact price fluctuations, it is important to think about possible outcomes. There may be times when the SPONGEV2 price increases due to factors such as greater adoption, utility expansion, and good sentiment. Price declines can also occur due to regulatory changes, changes in investor sentiment, or market corrections. With any SPONGEV2 outlook, investors should be prepared for both positive and negative possibilities and consider things like project progress, market trends, and overall economic conditions. To mitigate the effects of market volatility, it is advisable to diversify your crypto holdings. Year Possible high Possible low 2023 $0.00042442 $0.00037243 2025 $0.00192 $0.0009483 2030 $0.01476 $0.003586 What is Sponge Token V2? A new cryptocurrency called has just emerged, following in the footsteps of its predecessor, Sponge Token V1. It is a decentralized coin based on blockchain technology and is intended to bring value and benefit to its ecosystem. Sponge Token V2 Participation in governance decisions, facilitating transactions, and access to platform features are just some of the potential uses of SPONGEV2 within its ecosystem. The many possible applications include decentralized finance (DeFi) protocols, gaming, and decentralized apps (dApps). A larger number of people will be able to use and support the token if the current crypto community gets involved with the two different coins, which could cause $SPONGE to explode. However, anyone who wants to invest in or use SPONGEV2 should be confident with its goals and features. That's why you should read the white paper, the roadmap, and the official communication channels. So, before you decide to invest in SPONGEV2, as with any other cryptocurrency, you must carefully consider your investment objectives, your risk tolerance, and the results of relevant research. Factors That Influence the SPONGEV2 Course Several variables influence the value of Sponge Token V2 in the crypto market. This in turn affects its price. Market sentiment also influences the SPONGEV2 token, like all other cryptocurrencies. Increasing demand and prices are possible consequences of good news, developments, and endorsements. Low prices and demand usually appear from bad news, sentiment, or both. Staking, participating in P2E (play-to-earn), and using SPONGEV2 in decentralized apps (DApps) are all ways in which ecosystem utility influences demand and value. An imbalance between supply and demand could cause the price of SPONGEV2 tokens to rise as their scarcity and usage increase. The future price performance and growth potential of $SPONGEv2 will be heavily influenced by these and other variables. We would therefore like to go into a few special points in more detail. Planned CEX/DEX Listing Placing the Sponge Token V2 on controlled and is an important step that will impact its liquidity and pricing. decentralized exchanges A larger number of traders could have access to CEX listings, which could lead to an increase in demand and trading volume. To increase awareness of $SPONGEV2, popular exchanges could introduce trading pairs that include SPONGEV2. Alternatively, the ideas of decentralization and user autonomy can also be aligned with DEX listings. SPONGEV2 value within the decentralized ecosystem is increased by its availability on DEX platforms, which allow users to trade without intermediaries. Exchange listings are something SPONGEV2 investors and users should keep an eye on as they bring new trading opportunities and liquidity, which can impact the coin's price dynamics. Before trading SPONGEV2 on a particular platform, be sure to research the reliability and security features of each exchange. Tokenomics The development and sustainability of Sponge Token V2 are supported by a well-structured . SPONGEV2 makes it easy for users to transfer their assets from V1 to V2. This guarantees an easy change for current token holders. tokenomics mechanism Staking rewards, which allow users to obtain V2 tokens by staking their V1 tokens, make up a significant portion of the total token supply at 43.09%. This encourages users to get involved in the SPONGEV2 ecosystem and make a long-term commitment. A portion of the token allocation goes towards various goals such as $SPONGE's bridging fund (26.93%), marketing (7.50%), development (4.47%), P2E rewards (8%), and controlled exchange liquidity ( 10%). The overall vitality of the ecosystem is ensured by these allocations. Everything is aimed at promoting liquidity, awareness, and utility. Demand Through Social Media Signals The demand for and price of Sponge Token V2 is heavily influenced by social media. Twitter, Reddit, Telegram, and Discord are just some of the platforms where groups of $SPONGE fans and users gather. A positive community atmosphere, healthy participation, and helpful people can skyrocket SPONGEV2, which will attract more users and investments. Support and advocacy from prominent members of these groups can increase demand even further. But the opposite can also happen. Price fluctuations and lower demand can result in controversy and bad news. To gain useful insights into the general health of the token and expected price changes, it is recommended to monitor social media platforms and community sentiment. Because signals on social media are sometimes speculative or misleading, investors should exercise caution and verify information from multiple sources. Nevertheless, the dynamics of the crypto industry are heavily influenced by social media. Is Sponge Token V2 the Best Cryptocurrency in 2024? Do you want to buy SPONGEv2? Then we will guide you through the process step by step: Link your wallet Start by linking your Trust Wallet or MetaMask to the widget mentioned above. To buy $SPONGE tokens, this is the easiest method. On the other hand, Uniswap lists $SPONGE with contract address 0x25722Cd432d02895d9BE45f5dEB60fc479c8781E. Substitute V1 against V2 Click “Stake Now” to open the staking dashboard if you already own $SPONGE tokens. You can then stake your V1 tokens to get V2 tokens. You should know that staking V1 tokens is blocked forever. Buy $SPONGE Those who do not have $SPONGE tokens can purchase them with a bank card, exchange them for ETH or USDT, or do both. When you purchase $SPONGE through our widget, your V1 tokens are staked forever. You will also receive the same amount of V2 tokens as a starting bonus. Use your V1 tokens to collect V2 tokens Only if you use your V1 tokens can you receive V2 tokens before the official launch. As you stake more V1 tokens over a longer period of time, the number of V2 tokens you can collect increases. Conclusion The is an exciting offer for investors and fans alike. After looking at SPONGEV2's past performance, current market dynamics, and possible price movements, it is obvious that this coin is one to keep an eye on in the crypto market. Sponge Token V2 $SPONGEV2's potential to establish itself in the crypto world is high due to its excellent tokenomics, growing community, and ambitions to be listed on both controlled and decentralized exchanges. However, keep in mind your risk tolerance and proceed with caution when investing in the crypto industry. The future of $SPONGEv2 is still up for debate and discussion as we look to 2024, 2025, and 2030. When deciding how to invest in this ever-changing market, it is important that you do your research, keep an eye on the market, and perhaps even seek advice from a financial advisor. Disclaimer: This article should not be taken as investment advice. Since cryptocurrency is a highly volatile investment product, please conduct your own research before investing.

The Evolution of Sponge Token V2: An Analytical Perspective on SPONGEV2's Market Development

স্পঞ্জ টোকেন V2 এর বিবর্তন: SPONGEV2-এর বাজার উন্নয়নের উপর একটি বিশ্লেষণাত্মক দৃষ্টিকোণ

Die Entwicklung von Sponge Token V2: Eine analytische Perspektive auf die Marktentwicklung von SPONGEV2

La evolución de Sponge Token V2: una perspectiva analítica sobre el desarrollo del mercado de SPONGEV2

L'évolution de Sponge Token V2 : une perspective analytique sur le développement du marché de SPONGEV2

स्पंज टोकन V2 का विकास: SPONGEV2 के बाज़ार विकास पर एक विश्लेषणात्मक परिप्रेक्ष्य

Sponge Token V2 の進化: SPONGEV2 の市場開発に関する分析的視点

Sponge Token V2의 진화: SPONGEV2 시장 개발에 대한 분석적 관점

A evolução do Sponge Token V2: uma perspectiva analítica sobre o desenvolvimento do mercado do SPONGEV2

Эволюция Sponge Token V2: аналитический взгляд на развитие рынка SPONGEV2

Sponge Token V2'nin Evrimi: SPONGEV2'nin Piyasa Gelişimine Analitik Bir Bakış Açısı

Sự phát triển của Sponge Token V2: Góc nhìn phân tích về sự phát triển thị trường của SPONGEV2

Sponge Token V2的演变：SPONGEV2市场发展分析视角

Collected Papers on Analytical Psychology by C. G. Jung is part of the HackerNoon Books Series. Read this book online for free on HackerNoon! : Collected Papers on Analytical Psychology Title : C. G. Jung Author : Constance E. Long Editor : February 10, 2015 [eBook #48225] Release date : English Language TABLE OF LINKS EDITOR'S PREFACE TO SECOND EDITION AUTHOR'S PREFACE TO SECOND EDITION AUTHOR'S PREFACE TO FIRST EDITION CHAPTER I CHAPTER II CHAPTER III CHAPTER IV CHAPTER V CHAPTER VI CHAPTER VII CHAPTER VIII CHAPTER IX CHAPTER X CHAPTER XI CHAPTER XII CHAPTER XIII CHAPTER XIV CHAPTER XV About HackerNoon Book Series: We bring you the most important technical, scientific, and insightful public domain books. This book is part of the public domain. C. G. Jung (2015). Collected Papers on Analytical Psychology. Urbana, Illinois: Project Gutenberg. Retrieved https://www.gutenberg.org/cache/epub/48225/pg48225-images.html This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at , located at . www.gutenberg.org https://www.gutenberg.org/policy/license.html

Collected Papers on Analytical Psychology by C. G. Jung - Table of Links

Analytical Therapy

provides a notable advantage over — data becomes available to consumers faster. In the traditional ETL, you would not be able to analyze events from today until tomorrow’s nightly jobs would finish. These days, within minutes, seconds, or even . With streaming technologies, we no longer need to wait for scheduled batch jobs to see new data events. Real-time processing batch processing many businesses rely on data being available milliseconds Live dashboards are . updated automatically as new data comes in Despite all the benefits, to the overall data processes, tooling, and even data format. Therefore, it’s crucial to carefully weigh out the pros and cons of switching to real-time data pipelines. In this article, we’ll look at several options to with the and maintenance effort. real-time streaming adds a lot of additional complexity reap the benefits of a real-time paradigm least amount of architectural changes Traditional approach When you hear about real-time data pipelines, you may immediately start thinking about Apache Kafka, Flink, Spark Streaming, and similar frameworks which to operate a distributed event streaming platform. Those open-source platforms are best suited to scenarios: require a lot of knowledge - when you need to continuously ingest and process reasonably of real-time data, large amounts - when you anticipate and and you want to decouple their communication, multiple producers consumers - or when you want to own the underlying infrastructure, possibly ( ). on-prem e.g. compliance While many companies and services attempt to facilitate the management of underlying distributed clusters, the . Therefore, you need to consider: architecture still remains fairly complex - whether you have the , resources to operate those clusters - do you plan to process by using this platform, how much data - whether the added complexity is . worth the effort In the next sections, we’ll look at if your real-time needs don’t justify the and of a . alternative options added complexity costs self-managed distributed streaming platform Amazon Kinesis AWS realized the customer’s difficulties in managing . As a result, they came up with — a family of services that attempt to . By leveraging serverless , you can create a data stream with a few clicks in the AWS management console. Once you configured your estimated throughput and the number of shards, you can start implementing data producers and consumers. Even though Kinesis is serverless, you still need to monitor the message size and the number of shards to ensure that you don’t encounter any unexpected write throttles. message-bus architectures a long time ago (2013) Kinesis make real-time analytics easier Kinesis Data Streams In , you can find an example of a Kinesis producer ( ) sending data to a Kinesis data stream using a Python client, and how to continuously send micro-batches of data records to S3 ( ) by leveraging a delivery stream. my previous article source consumer/destination Kinesis Data Firehose Alternatively, to consume data from Kinesis Data Stream, we could: - aggregate and analyze data with , Kinesis Data Analytics - use Apache Flink to . send this data into Amazon Timestream The main benefits of using Kinesis Data Streams as compared to sending data directly to your desired application are and . Kinesis allows you to within the stream for up to and have that would receive data at the same time. This means that if a new application would need to collect the same data, you could add a new consumer to the process. This new consumer would not affect other data consumers or producers thanks to decoupling on the Kinesis architecture level. latency decoupling store data seven days multiple consumers Amazon Timestream As mentioned in the previous section, the . If you don’t need multiple applications that would regularly consume data from the stream, you could considerably streamline the process by using Amazon Timestream — a allowing you to analyze data in ( ) real-time. The underlying architecture is smart enough to for fast retrieval of real-time data, and then it automatically moves “old” data to cheaper long-term storage according to the specified retention period. major advantage of Kinesis is decoupling serverless time-series data store near ingest data first into an in-memory store Why would you use a time-series database for real-time data? Any new data record comes into the at a particular time. You may be tracking price changes over time, sensor measurements, logs, CPU utilization — practically . Therefore, it makes sense to consider using a time series database such as Timestream. The simplicity of the service makes it very appealing, especially if you would like to use SQL to retrieve data for analytics. stream any real-time streaming data is some sort of a time series When comparing the SQL interface of Timestream against the one available in Kinesis Data Analytics, Timestream is a clear winner. Kinesis SQL is quite obscure and introduces a lot of specific vocabulary. In contrast, Timestream provides an intuitive SQL interface with many useful built-in time-series functions, making time-based aggregation ( ) much easier. ex. minutely or hourly time buckets Side note: don’t use semicolons at the end of your queries in Timestream. If you do, you’ll get an error. Demo: Real-time ingestion into Timestream using Python To demonstrate how Timestream works, we’ll be sending cryptocurrency price changes into a Timestream table. Let’s start by creating a Timestream database and table. We can do all that either from the AWS management console or from AWS CLI: The above code should create a in your AWS region. Make sure that you use one of the in which Timestream is available. database regions Side note: The easiest way to find available regions for any AWS service is to check the pricing page: https://aws.amazon.com/timestream/pricing/ . Now we can create a . You need to specify your in-memory and magnetic store retention period. table Our database and table are created. Now we can get the latest price data from the . This AP provides many useful endpoints to get the latest information about a cryptocurrency market. We will focus on getting real-time price data for selected cryptocurrencies. Cryptocompare API We’ll get data in the following format: {‘BTC’: {‘USD’: 34406.27}, ‘DASH’: {‘USD’: 178.1}, ‘ETH’: {‘USD’: 2263.64}, ‘REP’: {‘USD’: 26.6}} Additionally, we need to convert this data to the proper Timestream format with a column, , and . Here is the full script that we can use to ingest new data every 10 seconds: . time measures dimensions https://gist.github.com/d00b8173d7dbaba08ba785d1cdb880c8 That’s it! The most time-consuming part is the definition of your dimensions and measures (lines 21–44). You should be about the : with Timestream you can only query data from a single table. No JOINS between tables are allowed. Therefore, it’s important to think ahead about your access patterns before you start ingesting data into Timestream. careful design of your measures and dimensions Here is how the data looks like in the end. Note that the ingestion time is presented in UTC: AWS Timestream: exploring the results in the query console — image by author We could now easily connect Timestream to Grafana for near real-time visualization. But that’s a story for another article. Never-ending script In the Timestream example above, running in a single process, we used a defined using while True. This is a common approach for a simple service ingesting data all the time, typically running as a background process or a service in a container orchestration platform. never-ending loop Minutely scheduled jobs An alternative to a continuously running script is a . The benefit of this approach is that it allows you to , which simplifies your architecture. You can think of it as a reversed Kappa architecture: while Kappa processes batch in the same way as real-time data ( ), this approach “batchifies” real-time data streams ( ) into micro-batches. service that is scheduled to run every minute treat this near real-time process as a batch job streaming-first approach batch-first approach Instead of while True, we now still ingest data roughly every 10 seconds but the actual process is executed once per minute, allowing us to track which runs were successful, and does not depend on the health of a single job run: There is no “right” or “wrong” approach. The main purpose of this method is to treat near real-time ingestion as a batch job. Here is a full Gist: . https://gist.github.com/d953cdbc6edbf8b224815cc5d8b53f73 Which option should you choose? The following questions may help you to make the right decision for your use case: - Which do you want to solve using real-time streaming: is it anomaly detection, alerting, product recommendation, dynamic pricing algorithm, tracking current market prices, understanding user behavior? Having a specific use case in mind can help you determine the right tool for the job, especially because there are a lot of specialized tools on the market. problem(s) - Which is acceptable in your use case? Is it OK if your data is available for analytics 1 minute after the event or stream has been received? Or on the contrary, you need a millisecond latency because otherwise, this data will no longer be actionable? latency - How many ( ) do you have to keep your platform operational? Does it make sense to spin up your own Kafka cluster, use some managed service, or maybe a serverless option such as Amazon Kinesis or Amazon Timestream can address your needs? resources employees and budget - How do you plan to and the health of your data streams? monitor observe - How much would be needed to teach your team how to use this specific platform? training - Which would need to be ingested in real-time, i.e. ? data sources data producers - What is the ( ) from which you would want to retrieve this data, i.e. ? And how do you want to retrieve this data — via SQL, Python, or perhaps only via analytical dashboards? target datastore data lake, data warehouse, specific database data consumers - In which way ( ) would you want to process this data? Is Kappa, , or other architecture worth considering to ? architecture-wise Lambda distinguish between real-time and batch Conclusion Ultimately, it depends on the problem that you try to solve using real-time processing technologies, the scale of your problem, and available resources. In many scenarios, a simple minutely batch job may be sufficient. It allows having a single architecture for all data processing needs, and data available within few minutes or even seconds after its generation. For other scenarios, Kinesis Data Streams or Amazon Timestream may provide simple yet effective means to add (near) real-time capabilities with very little maintenance effort. Lastly, if you do have employees who know how to operate Kafka, Flink, or Spark Streams, those can be helpful if you want to own your infrastructure and not being reliant on cloud providers. As always, thinking about the problem at hand will help assess the trade-offs and make the best decision for your use case. Also published on: https://dashbird.io/blog/real-time-processing-analytical/

Real-Time Data Processing for  Analytical Use Cases: Is it Worth it?

Emerging low-code development platforms enable Data Science teams to derive analytical insights from Big Data quickly. As organizations become proficient in capturing, storing, and analyzing data from multiple sources, they are discovering previously untapped business opportunities. This has been possible with the help of which has been enabling the companies to make smarter, data-driven decisions, as well as build & deploy Big Data solutions faster. The challenge, however, is that the same services are not yet available at the mid-sized or smaller companies often due to the lack of Data Science professionals. Data Science Enter Low-code development With graphical user interfaces and configuration, the Low-Code technology allows non-tech professionals to enter the world of development. They can build applications with no prerequisite knowledge of coding or other database management services. Gartner the global Low-Code Tech market to burgeon by 23% in the year 2021. forecasts How does this work for Data Scientists? platforms enable Data Science teams to derive analytical insights from Big Data quickly. With the co-existence of an array of features like Visual Modelling, Real-time monitoring & reporting, and Cross-platform accessibility among others, the low-code creates templates that replace any repetitive code structure, reducing the load from the algorithms. Low-code development This adds value to the work of developers and data scientists & accelerates the decision-making process. They can then focus on constructing information perceptions, structuring big data projects, or creating new products. Leveraging Low Code for Big Data Analytics: The data is still the data, but the ways of getting insights are continuing to improve. The use of Artificial Neural Networks like Machine learning in automating Big Data solutions has augmented exponential growth in the Digital economy. However, with a long & expensive deployment process, organizations are moving towards Low-Code programming for . Big Data Analytics With this, they needn’t rely on stressed data scientists for their acumen to extract insights from structured & unstructured data— to enhance business deftness. This synergistic relationship between the data scientists & those who aren’t programmatically inclined in the organization can foster more data work in a quicker amount of time without any compromise on data analysis. Furthermore, as much as Low-Code programming is ideal for non-tech professionals, it can also be used by data scientists to build data prototyping efficiently. Lastly, while low-code techniques may not replace traditional methods instantaneously, they can help less experienced as well as professional data scientists, in uncovering analytical solutions.

Low-Code Development Helps Data Scientists Uncover Analytical Insights

Preventing fraudulent transactions online is a cumbersome process for all the parties involved. With the digital payment ecosystem expanding each year, new technologies and companies develop innovative ways to detect fraudulent transactions in real-time. In an online ecosystem where trust between the parties is deficient - new-age technologies are waiting to disrupt the ecosystem. Artificial intelligence-based training algorithms are successfully flagging and resolving fraudulent transactions in real-time. I sat down with Nimrod Lehavi, Co-founder and CEO at Simplex, to discuss how new-age technologies are changing the digital payments ecosystem. Hi Nimrod, welcome to our “Behind the Startup” series. Tell us about yourself and the story behind Simplex? Ishan Pandey: I began my entrepreneurial career in the 90’s as a self-taught software developer. I’m a hacker at my core and I enjoy finding solutions to new and challenging problems. After 15 years of running my own software development business, solving complex business problems for many companies in various sizes and verticals, I stumbled upon The Bitcoin Whitepaper (the manifesto of Bitcoin, for any readers not familiar) and was charmed by it. No solution existed that allowed me to buy bitcoin with my credit card, and in general, it was constrained. I knew I wanted to do something new in this exciting young ecosystem, So I decided to make bitcoin accessible, in 2014 Simplex was born. Today I lead over a hundred of Simplex employees throughout the world as we work alongside our vast partner network to drive enablement and advance the ecosystem, helping anyone, anywhere, securely on-ramp, anyway. Nimrod Lehavi: How is fraud prevention different in crypto compared to processing payments for traditional credit card companies? Ishan Pandey: Fraud prevention differs between verticals, but it’s incredibly complex when it comes to crypto. The crypto space is unique in being truly global, 24/7/365 and that many who fall prey to fraud don’t even know it. It’s as much about identifying that users are aware of what they’re doing as it is weeding out real fraudsters. Fraud prevention warrants a lot of creative and analytical thinking always to stay two steps ahead of scammers, and it’s incredibly dynamic. We leverage a combination of skilled analysts and AI to detect all types of fraud at their earliest stages and annihilate them. Nimrod Lehavi: For new users, on-ramping is a confusing and complicated process. How is crypto on-ramping being democratized? Ishan Pandey: Excellent question. Crypto on-ramping has not traditionally been easy and I believe that the simpler we can make the process of on-ramping, the more the ecosystem will grow. On-ramping has been making steps toward democratization since we first pioneered methods for entering the ecosystem using credit or debit cards. We’ve now gone beyond credit or debit cards to enable uniquely large and flexible purchases across our partner network with our Banking solution. Leveraging Simplex Banking, our partners can allow users to on-ramp with any fiat anywhere globally, making giant leaps towards mass crypto adoption. Nimrod Lehavi: What problems do you see that still need to be tackled when it comes to Cryptocurrency On-Ramps and Off-Ramps? Ishan Pandey: Given the recent Bitcoin Boom, more users will be looking for a seamless way to off-ramp using rails that they are already comfortable with. The knowledge that it is just as easy to off-ramp, or utilize crypto earnings, as it is to on-ramp has been proven to increase the number of on-ramps in a given exchange. As a community, we need to continue working to ensure that crypto funds are easily accessible to the mass market. Nimrod Lehavi: Fraud prevention and management is a massive problem for exchanges, brokers, wallets, etc. What technologies and protocols are you deploying for preventing fraudulent transactions? Ishan Pandey: We are always vigilant to new fraud trends and strategies to protect our partners and their users. As fraud is an incredibly dynamic threat, we are continually expanding our fraud prevention technologies and protocols. An example of a protocol we employ is identifying user patterns and building a user story to see if the patterns of behaviour fit the user profile. By triangulating data and leveraging both advanced AI and seasoned analysts, we can build a profile of each user to make a timely, yet informed, judgment call, protecting our users and partners from fraud. Nimrod Lehavi: What new fraudulent and sophisticated techniques are you seeing being deployed by scammers? Ishan Pandey: Unfortunately, the panic surrounding COVID-19 launched a wave of fraud schemes. For example, some scammers encouraged the transfer of crypto funds in exchange for the sale of masks (which they never received), others included fraudulent CDC emails that facilitated the transfer of funds to invest in a COVID cure, among others. Ponzi schemes involving cryptocurrency amounted to billions last year, and scammers are moving beyond email to use social media platforms, such as YouTube, Instagram, and Facebook, to promote their scams. When faced with an email or request, I encourage users to always look into a company before working with it and do their due diligence to make sure they are familiar with crypto concepts and what is being asked of them. Nimrod Lehavi: What advice would you give to a startup that wants to put measures for fraud detection and prevention in place? Ishan Pandey: It’s essential to have a well-prepared and trained team who are well-versed in all there is to know about fraud types and prevention. It’s also crucial to have up-to-date and sophisticated protective measures and checks in place to ensure that you are always one step ahead of the scammers. But what makes the difference is developing a profound understanding of your market, and putting yourselves in the shoes of the other side, trying to “think like the bad guys” and preempt the scams before they occur. Nimrod Lehavi: Do you think that being based in Israel, which is a global leader in cybersecurity, gives you a competitive edge as compared to others? Ishan Pandey: Being based in Israel is part of our DNA. As a company born out of the need to make crypto accessible, that mission continues to underscore and lead the company. In true, StartUp Nation fashion, we pride ourselves on being a talented and passionate group that works together to create unique solutions for difficult problems. Our team comes with a vast background in cybersecurity, finance, and defence, and brings years of quick and decisive problem-solving to join together and create something meaningful. However, the diversity of our team and values of problem-solving are not limited to our Israel office, and this culture permeates throughout our office in Europe as well. Nimrod Lehavi: What technology and trends are you looking at, which can disrupt our society? Ishan Pandey: I believe crypto has a lot to offer to the world: a lasting impact on our financial systems hence on society, a real change in the infrastructure of the economy and financial services. Our goal always has and will always be to democratize crypto accessibility and adoption for everyone. Nimrod Lehavi: I see the crypto ecosystem as a potential alternative financial ecosystem; fast, innovative, secure, and transparent. If indeed globally accessible, think about how that can impact the world. No matter the regime one may find themself in, they will be able to take hold of their finances and not be subject to others’ whims. From our Banking solution, all of our products to our Principal Membership with Visa are geared to enable our partners to offer anyone, anywhere access to purchase, anyway. As we enter into 2021, we will be focusing our efforts on making crypto even more accessible to anyone, anywhere, throughout our trusted partner network. The purpose of this article is to remove informational asymmetry existing today in our digital markets by performing due diligence by asking the right questions and equipping readers with better opinions to make informed decisions. The material does not constitute any investment, financial, or legal advice. Please do your research before investing in any digital assets or tokens, etc. The writer does not have any vested interest in the company. Interviewer - Ishan Pandey.

Fraud Prevention Requires Creative & Analytical Thinking To Stay Two Steps Ahead - Nimrod Lehavi

This is a hands-on course on how to deploy a fully Serverless web app using the . You will learn how to: AWS CDK Structure **CDK Stacks **to deploy an application from end-to-end Deploy a integrated with for dynamic requests processing Store data in a fast and cost-effective way with REST API AWS Lambda DynamoDB Use as a source for in an event-driven architecture DynamoDB streams Lambda Ingest and manipulate loads of data streams with Kinesis Firehose Deploy and query a with , and your entire application health in a single place using Data Lake Athena S3 GlueMonitor Dashbird You can use the resources declared in this demo application as a starting point to mix and adapt to your own architectures later, which should save you quite some time. The App and Architecture The demo app is a public blog where anyone can read, publish and like posts. It’s available . Go ahead and publish something in the top-left corner (yellow button) and also “like” articles already published. Check out the codebase on . on this link this repo Frontend Backend Data Lake and Analytical Querying What is the AWS CDK? CDK stands for Cloud Development Kit. Think of it as CloudFormation (CF) in your preferred language (Python, Typescript, C#, etc). Roughly speaking, it works like this: 1. You declare cloud resources using classes provided by the CDK libraries. Example: aws_cdk aws_s3
my_bucket = aws_s3.Bucket(self, ) from import 'MyBucket' 2. Run CDK deploy 3. CDK translates this to a CloudFormation template and deploy it on AWS for you. In case you would like to dig deeper, that will get your basics started. I also strongly recommend reading the . AWS also has a workshop official CDK documentation Advantages using the AWS CDK Use languages that are more expressive than YAML or JSON, for instance. Less - much less! - verbose than CloudFormation templates. Easier to apply reusability and inheritance principles to infrastructure code. Better integration with IDEs for code completion, IntelliSense, etc. Possible to test your infra code, just as any other software Portable: since it’s just a wrapper around CF, we can easily port it to JSON or YAML Disadvantages of using the AWS CDK Although released as a stable project by the AWS team, many parts (a lot of the good ones) are still experimental and APIs may change in backwards incompatible ways. It’s under constant development. During the preparation of this course, I had to upgrade my libraries three times. Documentation is still lacking at some parts and you will need to look at the CDK code, occasionally, to understand how to declare certain things. Deploy it yourself Although we have provided an online demo, you can also deploy this app in your own AWS account: Clone the repo: git clone git@github.com:byrro/serverless-website-demo.git sls-demo; cd sls-demo Setup your virtual environment: virtualenv -p /urs/bin/python3.8 .env; source .env/bin/activate; pip install -r requirements Specify an AWS account ID: ** export AWS_ACCOUNT_ID=1234567890 Deploy all three stacks: cdk deploy sls-blog; cdk deploy sls-blog-api; cdk deploy sls-blog-analytical ** You can also hard code your Account ID in the CDK project, as I’ll show in a minute; When starting a new project from scratch, you would run . This is not necessary for this demo, since the project is already created. cdk init --language [python|typescript|...] Monitoring Deploying this architecture in the cloud and blindly believing it will work flawlessly is not reasonable. We want to be the first one to know when something is not right to act upon it as quickly as possible. In this project, I used for its ease of use and seamless integration. Instead of having to deploy an agent inside my code, Dashbird plugs into my Stacks that I can deploy with the effort of one click. It not only monitors Lambda function errors, but also other resources that we’re using, such as DynamoDB tables. They even suggest cross-referenced against industry best practices. Dashbird through a CloudFormation template insights for architectural improvements Finally, Dashbird offers a free-forever plan. It’s a no-brainer to try it out by . registering for free How a CDK project is structured A CDK project creates an “Application”. This app may have one or more “Stacks”. A Stack is a group of cloud resources (Lambda functions, S3 buckets, etc) that are instantiated using CDK classes. It’s also possible to have multiple applications in a single CDK project. App Object Creating a CDK app is as simple as: app = core.App() When you run , an initial application with basic boilerplate code is created for you in the project root, under . cdk init --language [language] app.py The next thing we need is an environment, which is composed of an AWS Account ID and Region: env = env = core.Environment(
    account= ,
    region= ,
) 1234567890 'us-east-1' Declaring an environment is not required (CDK can infer from your AWS credentials), but is a good practice. Most of us work with multiple AWS accounts. It’s easy to mess around with several projects, accounts, credentials. When we explicitly set the environment in the CDK app, it’s locked and prevents mistaken deployments. Now we declare our stacks: my_project.my_project_stack MyStack

my_stack = MyStack(
    app, ,
    env=env,
) from import 'my-stack' This is how we our stacks for deployment. In the next section we’ll see how to those stacks. instantiate declare Stack Object The Stack object is where we our AWS resources. It inherits from the core.Stack CDK class and accepts a scope - which is our app object - a string identifier and an environment. declare ( . ): ( , : . , : , : . ,
            ** ,
            ) -> : (). ( , , ** )

        # class MyStack core Stack def __init__ self scope core Construct id str env core Environment kwargs None super __init__ scope id kwargs Declare AWS resources here Declaring AWS resources To declare AWS resources, we need a specific library for each service. Here’s a list of all and their counterparts. Other flavors are and . Python libraries Typescript Java .NET Let’s see how a basic REST API would be declared (typing expressions were removed for readability purposes): aws_cdk aws_apigateway, aws_lambda = aws_lambda.Function(
            self, ,
            runtime=aws_lambda.Runtime.PYTHON_3_8,
            code=aws_lambda.Code.asset( my_lambda.handler sls-blog-rest-api-gateway from import ( . ): ( , , , ): (). ( , , ** ) class MyStack core Stack def __init__ self scope id env super __init__ scope id kwargs my_lambda 'MyLambda' 'my_lambda_folder),
            handler=' ',
        )

        aws_apigateway.LambdaRestApi(
            self,
            ' ',
            handler=my_lambda,
        ) We first declare a Lambda function . We point its code to the . Inside this folder, there should be a file, containing a function called . This handler function should accept Lambda invocations normally (an and objects). my_lambda my_lambda_folder my_lambda.py handler event context Next, a is declared, using as the handler (not to confuse with the Lambda’s handler function). This will create a new API Gateway REST API integrated with using an . All HTTP requests will be routed to the Lambda function. LambdaRestApi my_lambda my_lambda AWS_PROXY integration type Our Project App & Stacks comprises one application with three Stacks. They’re all declared in the and files. This project app.py sls_website_stack.py Below we’ll walk through all Stacks in a high level. I encourage you to inspect the to learn how these resources are declared and also integrated. For example: a Kinesis Firehose is created in one Stack and referenced in another to include its name as an environment variable for the Lambda function that will interact with it. stacks file Except for the frontend static Stack - which is small - you will notice that resources are initialized with a (null) value in the beginning. The reason is that, even though the CDK is generally more succinct than CloudFormation, it can still be lengthy enough to clutter the view of the entire Stack. Having each resource declared first in one line, I can provide a short summary of everything that’s in the Stack and then instantiate the CDK classes in other methods. None = static_stack

        # SQS Queues
        self.queue_ddb_streams_dlq = None  # Dead-letter-queue DDB streams

        # DynamoDB Tables
        self.ddb_table_blog = None  # Single-table all blog content

        # DynamoDB Event Sources
        self.ddb_source_blog = None  # Blog table streams source

        # DynamoDB Indexes
        self.ddb_gsi_latest = None  # GSI ordering articles by timestamp

        # Lambda Functions
        self.lambda_blog = None  # Serves requests to the blog public API
        self.lambda_stream_reader = None  # Processes DynamoDB streams

        # Continues other resources... ( . ): ( , , , , ): (). ( , , ** ) . class SlsBlogApiStack core Stack def __init__ self scope id env static_stack super __init__ scope id kwargs self static_stack for for with Notice it takes another Stack object ( ) as an argument to its initialization. In the file, you can see that the is initialized passing the as an argument. static_stack app.py SlsBlogApiStack SlsBlogStack We use it to reference the CloudFront distribution domain ( ) in the Lambda environment variables. This variable can be used to customize the HTTP response header to comply with . This illustrates one way to easily integrate and reference information from one Stack into another within a CDK project. d1qmte5oc6ndq5.cloudfront.net Access-Control-Allow-Origin CORS standards At the end of the initialization, another method is called to instantiate the CDK classes for each resource and configure their parameters. self.create_cdk_resources() Next we’ll walk through each of our project’s Stacks. Stack 1: static website Our focus is on the Serverless backend, so the frontend here is terribly rough and simple. It’s stored in an S3 Bucket and distributed through a CloudFront CDN. CDK has a helpful class called BucketDeployment. It takes the contents of a directory and sync to an S3 bucket. In this case, we stored the frontend code in the folder. website_static aws_s3_deployment.BucketDeployment(
    self, ,
    sources=[aws_s3_deployment.Source.asset( )],
    destination_bucket=static_bucket,
    distribution=cdn,
) 'SlsBlogStaticS3Deployment' 'website_static' Stack 2: API/backend Our backend consists of an API Gateway (REST) connected to a . Microservices receives a lot of press, but you probably shouldn’t always break your applications in several functions. A Monolith is just fine - -, really. monolithic Lambda function and sometimes recommended This API & Lambda support a single endpoint (with GET and POST methods) with a queryString “action”, which takes three parameters: : populates the latest blog posts get-latest-articles : triggered when someone likes an article like-article : post a new blog article publish-article Here’s the power of the CDK model. We can create a REST API with 10 lines of code: rest_api_blog = aws_apigateway.LambdaRestApi(
    self, ,
    handler=lambda_blog,  # Previously declared Lambda 'sls-blog-rest-api-gateway' = . ( ),
    ),
) function deploy_options aws_apigateway StageOptions stage_name= ,
        throttling_rate_limit=lambda_param_max_concurrency,
        logging_level=aws_apigateway.MethodLoggingLevel( 'api' 'INFO' One nice little thing is that Lambda memory is used as a cache for the latest articles. We load the cache container outside the Lambda handler function. It remains in memory even after an invocation ends and is available for subsequent requests. Learn more here about . how to use Lambda as a cache mechanism MAX_CACHE_AGE: int = # In seconds
CACHE_LATEST_ARTICLES: Dict[str, Union[int, list]] = { : time.time(), : [],
} 120 'last_update' 'articles' All the data is stored in DynamoDB (DDB) using a , in . The site only shows the latest blog articles and items get after a few days by setting a attribute. single-table design on-demand mode auto-deleted by DDB time-to-live ddb_attr_time_to_live = ddb_param_max_parallel_streams = ddb_table_blog = aws_dynamodb.Table(
    self, ,
    partition_key=aws_dynamodb.Attribute(
        name= ,
        type=aws_dynamodb.AttributeType.STRING,
    ),
    billing_mode=aws_dynamodb.BillingMode.PAY_PER_REQUEST,
    point_in_time_recovery=True,
    removal_policy=core.RemovalPolicy.DESTROY,
    time_to_live_attribute=self.ddb_attr_time_to_live,
    stream=aws_dynamodb.StreamViewType.NEW_AND_OLD_IMAGES,
) 'time-to-live' 5 'sls-blog-dynamo-table' 'id' The DDB table also has a that makes it easier to retrieve articles ordered by date for the site: GSI (Global Secondary Index) self.ddb_table_blog.add_global_secondary_index(
    index_name= ,
    partition_key=aws_dynamodb.Attribute(
        name= ,
        type=aws_dynamodb.AttributeType.STRING,
    ),
    sort_key=aws_dynamodb.Attribute(
        name= ,
        type=aws_dynamodb.AttributeType.NUMBER,
    ),
    projection_type=aws_dynamodb.ProjectionType.ALL,
) 'latest-blogs' 'item-type' 'publish-timestamp' Modifications to DDB items generate streams that are processed by a . These streams are then repackaged and sent to a Kinesis Firehose stream processor. second Lambda function DDB doesn’t provide the flexibility that SQL databases offer and many choose , for example. Although Aurora is a great service, personally I prefer DDB for its simplicity and reliable, consistent performance. But sometimes we do need to run analytical queries, those with aggregations and on-the-fly filters. For that, we’ll be using Athena (more in the next Stack). Aurora Serverless Stack 3: analytical querying A Kinesis Firehose Stream is responsible for batching data inserted/modified in DDB, converting them to Apache Parquet format and storing in dedicated S3 buckets. From S3, we create a Data Lake with AWS Glue (used to declare our data schemas) and Athena (used to query the data). Athena is extremely powerful. We can statements ( ) to query terabytes of data and pay on-demand ($0.005 per GB of data scanned). Using Parquet not only improves query speed, but also reduces cost by minimizing the amount of data Athena needs to scan for each query. use SQL SELECT with some limitations Queries that are impossible or expensive/slow in DynamoDB, such as aggregations and JOINs, are fast and cheap in Athena. The two services combine each other in a perfect way, so that your application has optimized transactional storage and flexible analytical querying capabilities. We can use Athena to query all articles ever published and cross-reference with likes and HTTP metadata (source IP address, country, device type, etc). Even articles that were already expired by DynamoDB TTL (time-to-live) would continue to be available in the Data Lake. For example, which countries are liking the most articles? In the AWS Console, we get something like this: Queries can also be executed programmatically with or (e.g. ) to integrate anywhere we need this data. Athena API AWS SDKs Python’s boto3 Athena also supports JOINs. Here’s an example joining articles and HTTP metadata to analyze the most popular authors among readers of a particular country: Deployment CDK can deploy one Stack at a time. Since we have three, it’s necessary to specify which one when running the command. We do that by passing in the Stack ID as a CLI argument. For example, the following command will deploy the SlsBlogApiStack (id: ): CDK deploy sls-blog-api cdk deploy sls-blog-api Since all Stacks involve some type of permission granting, CDK asks for confirmation before deploying those resources. You can review the permissions requested and hit y when it’s good to go. Wrapping up We’ve covered how to structure CDK apps and add a bunch of AWS Resources to deploy with a simple command. CDK deploy If you’re new to the CDK - and as suggested early in the article -, it’s strongly recommended to follow AWS and . CDK workshop documentation Keep an eye on future publications as well, as is releasing other examples and tutorials to reap the most out of AWS serverless services with the power of infra automation with CDK or else. Dashbird Previously published at https://dashbird.io/blog/crash-course-aws-cdk-serverless-rest-api-data-lake-analytical-querying/

Stories, News & Mentions about ANALYTICA

Analytica’s Stories on HackerNoon

Analytica's News & Mentions