Overview
Time flies and another year passed. Today, I’d like to share with you my year 2019 as a software engineer. The fun, the hard, the unexpected, and the amazing parts. In this article, I will talk about my working experience at Nuxeo and Datadog. Then, share my open-source experience at VAVR and end with the blogging I’ve been doing for the whole year.
- Working at Nuxeo
- Working at Datadog
- Open-source project: VAVR
- Blogging
Let’s get started!
Nuxeo
Nuxeo is a content management platform to build modern business applications. I work at Nuxeo Online Services (NOS) Team, a SaaS (software as a service) application including all online services of Nuxeo: Nuxeo Studio, Nuxeo Marketplace, Nuxeo Connect Dashboard, Nuxeo Register, and command-line tool “nuxeoctl”. The main programming language used by Nuxeo is Java. This year, I spent 10 months at Nuxeo before leaving the company. At Nuxeo, I worked on improving the versioning system based on JGit, adding RESTful API, creating Docker image, and solving performance bottlenecks. There were many backgrounds works as well, such as bug-fixing, upgrading legacy frameworks, CI improvement, release, deploy, operations and monitoring of our production. The work is challenging since we are a small team (1 PO + 1 designer + 2 front-end engineers + 3 back-end engineers).
I wrote another article, Nuxeo 2019, which explains more in detail about what I did this year at Nuxeo over plan, code, build, test, release, deploy, operation and monitor. You can check that if interested :) After working 3 years at Nuxeo, I chose to leave for another challenge: I joined Datadog.
Datadog
Datadog is a monitoring service for cloud-scale applications, providing monitoring of servers, database, tools, and services, through a Saas-based data analytics platform. I joined the Logs Hot Storage Team at Datadog in November 2019. Our team owns the logs indexing and storage underpinning the Datadog logs management application. We build, scale, and operate some of the high-volume data systems that power the growth of Datadog’s business.
Image: Datadog
In the last two months, besides the onboarding process, I mainly worked on two topics: data export and logs filtering. Let me explain a bit in the next paragraphs.
Data export. For internal analytics purposes, we need to export data from our internal system to AWS S3. It requires some knowledge about AWS such as S3 bucket policy, IAM role. The most tricky part is testing because I don’t have permission to perform the changes myself. Therefore, it requires a cross-team collaboration to make it work.
Logs filtering. Datadog stores your logs in different indexes via index filters (documentation). Index filters give dynamic control over what goes into your indexes. However, you may not want to save 100% of your logs. You may only want some of them for troubleshooting purposes. In this case, you can define an exclusion filter to exclude a given percentage. The exclusion is based on message ID. Currently, I’m working on a new feature that allows you to exclude based on other criteria. For now, I cannot give more detail now but it’s an interesting feature. In regards to implementation, it modifies the data models in RESTful API and persistence; it requires architectural changes; modification on hashing and sampling algorithm; benchmark about performance since millions of logs passed every second; and thinking about deployment (how to lower the risk for customers in case of incidents).
Vavr
VAVR (formerly called Javaslang) is a non-commercial, non-profit
object-functional library that runs with Java 8+. It aims to reduce the lines of
code and increase code quality. It brings different Scala concepts into Java such
as control structures (Option, Either, Try, …) and purely functional
collections (List, Set, Map, Array, …). They are immutable,
type-safe, and interchangeable with builtin Java types. This library makes
exception handling and Java collection usage easier, more concise, and more
robust.
This year, I contributed mainly to VAVR Core and VAVR Jackson to help the community. I first started writing blog posts and using VAVR myself. Then I continued on improving documentation and now handle bug-fixing. The most interesting part of supporting open-source software is the chance to work with other engineers outside of your company and see how other code-base works. It is also nice to see your work being used by other developers.
- Blog: VAVR List vs Java List (link)
- Blog: VAVR HashMap vs Java HashMap (link)
- Core: Add simple examples to Javadoc (link)
- Core: Improve shuffle test (link)
- Core: Fix Integer overflow: Stream.rangeClosedBy returns empty stream for big numbers and negative step (link)
- Core: Vector.ofAll(someListView) shouldn’t perform a linear copy. (link)
- Jackson: Cannot deserialize to Map<String, String> (link)
- Jackson: Cannot deserialize/serialize YearMonth / LocalDateTime with specific @JsonFormat (link)
- Jackson: Check @JsonFormat support in LazySerializer and LazyDeserializer (link)
- Benchmark: Recover the broken benchmark execution (link)
If you were interested in using Vavr, you can visit the official website, read the Vavr User Guide, or chat with the community at Gitter. Actually, I also contributed to other communities, such as Checkstyle, Debezium, Apache Commons CLI. Due to time constraints, I cannot share more here.
Blogging
This year, many of you visited my blog. There are 149k users with 187k page views. The traffic has increased by more than 600% compared to 2018. This is a huge accomplishment: I feel honored to share my technical journey with you and bring added value to your daily work. In 2019, 41 blog posts have been written: mainly focus on Java, Maven, Testing, and Git. My secret about blog writing is to keep delivery at a fixed pace: writing a blog post every week or every two weeks. It will make a big difference in the long term.
However, the existing model is not perfect. There are many improvements to do. For example, compared to 2018 (orange), the average of session duration in 2019 (blue) dropped significantly. In 2020, I plan to write fewer articles and keep improving the existing one, so that they will be up-to-date. In other words, they won’t be “one-shot” production but will be maintained over time.
Conclusion
In this article, I shared briefly my working experience at Nuxeo and Datadog in 2019 as a software engineer working for SaaS applications, my contributions for VAVR, a famous object-functional library for Java 8+, and finally, some analytics about this technical blog. Interested to know more? You can subscribe to the feed of my blog, follow me on Twitter or GitHub. Hope you enjoy this article, see you the next time!
References
- Datadog, “Indexes”, Datadog Docs, 2019. https://docs.datadoghq.com/logs/indexes/
- Vavr, “Home Page”, Vavr, 2019. https://www.vavr.io/