Articles

The latest in Cleareye.ai news and industry updates.

Macro Technology Trends Adopted by our Product Engineering Team in 2020

Technology is evolving so quickly that anyone with the title Architect can feel overwhelmed when it comes to deciding what to choose and how long the choice is going to last. This isn’t going to change anytime soon, and here are some of the technology and process components that we adopted to production from multiple experiments that we ran this year.

Hybrid Data Model

JSON being the de facto standard for REST and microservices, we depend a lot on document-oriented stores like Apache CouchDB. While some data is best modelled as rows, other data is best modelled as JSON documents. So we started moving to hybrid data models(relational+JSON). MariaDB’s JSON type and comprehensive set of SQL functions for storing, indexing, and querying JSON documents were adopted.

Explainability

Bringing explainability to the data we consume and the model predictions. Adoption of Graph database (Neo4j) gave a context on what features and weights were used in predictions for our talent matching engine and gave the customer confidence to give the go-ahead for production. Graph Algorithms were adopted to our reputation risk solution, which gave the context for a risk alert.

Declarative data pipeline definition

Traditionally data pipelines were written in Python scripts, we were no exception. The script contains the logic of the individual steps as well as the code chaining the steps together. As we started building more complex data engineering pipelines, maintenance & reliability of the pipelines became a challenge. This year we started building declarative data pipelines for our data lifecycle; from ingestion, to clean up, to transformation leveraging Apache Nifi. With declarative data pipelines, data engineers can now rapidly branch, edit, and deploy data pipelines with existing DevOps tools for CI/CD. As we end the year, we have a stable data pipelines build and run using Apache Nifi.

Continuous delivery for machine learning (CD4ML)

CD4ML is a term proposed by Thought Works, which is beyond the much talked about term MLOps. CD4ML talks not just about version models, also about training data, hyperparameter. We decided to build our large scale AI/ML pipelines rely on Apache MinIO as the model and data store. MinIO was the choice considering the performance characteristics and the ability to seamlessly scale to Petabytes and Exabytes of data.  By separating storage and compute, we built a custom framework that is not dependent on local resources, but runs on a container inside Kubernetes. This adds considerable flexibility to our deployment and meets the demand of scalability.

Other experiments that moved from trial to adoption are: the change from Nginx as our reverse proxy to Traefik. Nginx has not lost its charm, but it didn’t meet our monitoring and observability needs, protocol support, and ease of use. Traefik is a Docker-aware reverse proxy that includes its monitoring dashboard. Traefik supports automatic discovery of services, metrics, tracing, and has Let’s Encrypt support out of the box

Streamlit turned out to be a good pick of the year. Streamlit is a framework for building ML and data science web apps. It was quick and fun building prototypes using Streamlit.

What we still have in the assess phase and did not make to adoption this year is the micro frontends. A couple of POC’s are built and are being tested out, expecting to hit our mainstream adoption by Q1 2021.


About the Author

Chandrasekhar Somasekhar is the Chief Architect at Cleareye.ai. He is responsible for product development, strategic technology direction, implementing and governing solution architecture methodologies at Cleareye.ai. Chandra defines and governs the enterprise solution architecture strategy. Additionally, he provides architectural direction. He is passionate about developing, mentoring, and motivating a high-performance team.

More Articles

Trading of goods in international market is complex and fraught with risks and uncertainties. Transactions are made between entities in distant countries and there is often a trust deficit between sellers and buyers. And this is why risk-mitigating trade finance products…

For the past few months, America has been waiting for a new COVID relief package. On Dec. 20, negotiators finally agreed upon a $900 billion package with $320 billion going towards the PPP (Paycheck Protection Program) for small business. The…

“If lenders don’t invest in robust technology solutions for managing the coming onslaught of loan applications, they risk, at best, falling behind other lenders and, at worst, mismanaging loans and applications when they do come in.” Much needed relief is…

The current economic crisis along with restrictions related to pandemic has significantly impacted small businesses. The Paycheck Protection Program (PPP) is a loan designed to provide incentive for small businesses to retain workers on the payroll. Small Business Administration (SBA)…

Technology is evolving so quickly that anyone with the title Architect can feel overwhelmed when it comes to deciding what to choose and how long the choice is going to last. This isn’t going to change anytime soon, and here…

The pandemic has hit businesses hard. With so many in need of assistance, the federal government and SBA are offering low interest and forgivable loans to small business owners. The PPP (Paycheck Protection Program) is one of the most popular…

According to the world economic forum more than 25% of a company’s market share is directly linked to its reputation. Reputation matters to all organisations, but very few have a plan to address reputation risks. There is no industry standard…

In today’s world, people are looking for a way to make their financial transactions as streamlined as possible. That is one of the reasons open banking is becoming so popular in the modern financial world.  This article will take a…

Major changes are coming to the finance industry worldwide as regulators are pushing for banks to move away from LIBOR (London Interbank Offered Rate). This is potentially the largest ever fundamental change that the finance industry has had to deal…

LIBOR is London Interbank Offered Rate, the benchmark rate in nearly $350 Trillion worth financial products contracts worldwide. Financial institutions have to completely transition away from LIBOR by the end of 2021. SOFR (secured overnight financing rate) has been developed…

In The News

The very latest in Cleareye.ai news.