2021-09-14 Problem The issue with Spring Cloud AWS Messaging and its @SqsListener annotation is that it cannot process more than 10 concurrent messages. Why is that? AWS SQS works via a polling model. A client is making requests to the ... Read More »
Writing Snowflake queries in practice
2021-08-23 At Conductor, we deal with pretty big data sets and provide users with a rich set of data views, filtering, and aggregations – all in real-time! Snowflake is a modern data warehouse that we use to power some of ... Read More »
What Is Technical Debt? How to Manage (and Reduce) Tech Debt
What Is Technical Debt? Technical debt is pretty much any code anyone wrote last week. It’s all around us, binds us, and basically duct-tapes-and-glues the tech world together. Tech Debt is a part of life and, while we all want ... Read More »
How to Use React Storybook to Create Reusable Components
Nowadays, user interface design is a costly process that requires effort from a lot of people, including developers, designers, testers, product managers and so on. Companies usually create interfaces consisting of many independent components that can be reused. At the same time, ... Read More »
Создание библиотеки компонентов с использованием Storybook
Введение В наши дни разработка интерфейса является достаточно затратным процессом который требует усилий от множества людей, разработчиков, дизайнеров, тестировщиков продукт менеджеров и так далее. Компании все чаще прибегают к написанию интерфейса состоящего из множества независимых компонентов, которые могут быть ... Read More »
The Power Of Kibana in 6 Simple Steps
At Conductor, our systems generate over 10 million lines of logs every day. On a particularly busy day, we can break 30 million lines. When the DevOps team started using the ELK stack (Elasticsearch, Logstash, and Kibana) to help us ... Read More »
Revamping Continuous Integration and Delivery at Conductor
As companies, products and teams evolve so do development practices. At Conductor we frequently identify areas for opportunity when working on Searchlight. Over the years we have made leaps and bounds in the way we develop, test and deliver high ... Read More »
Using Java 8 Streams to Process Large Amounts of Data
Today, we’re open sourcing an in-house Java 8 stream utility library, which can aggregate, merge, or join streams in a memory efficient way. To skip right to the code, check out the github repo. Here at Conductor, we build Searchlight, a ... Read More »
Bulk ETL from Hadoop to Cassandra
Cassandra’s distributed architecture enables a high write throughput compared to other NoSQL and SQL database systems. Most benchmarks focus on incremental writes, but data can also be efficiently ingested by Cassandra in bulk. Bulk loading is an attractive option when ... Read More »
How to Build a Speedy Custom Compression Codec for Hadoop
What’s Data Compression? Data compression provides a means by which a system can improve I/O performance by shrinking the data being transmitted between stores and processes. Anyone who uses the Internet knows that downloading an immense file takes noticeably longer ... Read More »