data // software // tech

The first three chapters of Atomic Habits discusses achieving goals and thinking about results, processes, and identities. That is, many people start with outcomes in mind when setting up for achieving goals. I want to run a marathon. Or, I want to make more money. The way to actually achieve these sorts of outcomes is to think about the identity of an individual who has more money or who runs marathons. If I want to stop eating garbage food so frequently I need to think about the identity...

I've always hated recursive logic because it makes my brain hurt. This is relatively straightforward approach to flattening a nested dictionary by recursively passing the value of a dictionary back to itself if it is itself a dictionary. d = { 'a':1, 'b':2, 'c':{ 'a':1 } } def flatten(d, parent_key=''): output = [] for k,v in d.items(): new_key = k + parent_key if isinstance(v, dict): output.extend(flatten(v, new_key).items()) ...

(this blog post is auto-generated via OpenAI) Here is a guide on how to become a data engineer: What are the Requirements for Becoming a Data Engineer? The requirements for becoming a data engineer vary depending on the company. However, most companies require that you have experience with SQL, Python, Java, or Scala. You should also have experience with Hadoop or Spark. You should also have experience with machine learning tools such as TensorFlow or Keras. You should also have experience wi...

(this blog post is auto-generated via gpt-neo) Using airflow to orchestrate your data pipelines In many organizations, data pipelines are used to transfer data with as little manual intervention as possible to keep the time to value of the process as low as possible. The following video demonstrates how to set up and orchestrate a data pipeline on AWS with AWS CloudFormation. Note It’s important to pay attention to AWS documentation when setting up or updating any AWS resource. For example, ...

(this blog post is auto-generated via gpt-neo) The best laptop for business, gaming, media etc. Let us break it down. So you have been looking at laptops so long, just what is it that you needed to buy? I want you to think again and take it from the top down. For a while, I have been looking at the laptop market, a lot like how people look at cars. And the same sort of “Top 10” type of question arises. What is the perfect machine for me to use. And, then what is the perfect machine for others...

thought Sometimes I feel like I need to say something in order for common courtesy to take place; even if I don't have anything to say or don't want to say anything. This can't be at all unusual, right? Anyway, it seems forcing it ends up with me saying things that make little sense since they're so forced. But to say nothing is often considered rude. So is it better to just say anything and risk just coming off aloof? bought Microsoft Surface Laptop 4 was announced today. I impulsivel...

I'm getting more and more familiar with GCP and its capabilites as I deploy applications through things like Cloud Run now. This has been sort of a gateway into me wanting to learn about the underlying infrastructure behind it, which has in turn led me to start learning about Kubernetes setups. I think one difficult thing about all of these options is that there are so many, that you have to really understand all of them in order to make a decision on which one you should be using. I think ...

Advancements in AI have brought a lot of attention to a number of subdomains within this vast field. One interesting one is natural language processing. What is a Hugging Face Transformer? Why don’t we let their pretrained models answer this question? Transformers provides general-purpose architectures for Natural Language Understanding (NLU) and Natural Language Generation (NLG) with over 32+ pretrained models in 100+ languages . The library currently contains PyTorch, Tensorflow and Flax im...

Standard Notes & Listed as a blog This is my first post using Listed.to and Standard Notes. Let's see how it works! def hello(): print("Hello World") if __name__ == '__main__': hello() ...

A short while ago I published a rather technical post on the development of a python-based attribution model that leverages a probabilistic graphical modeling concept known as a Markov chain. I realize what might serve as better content is actually the motivation behind doing such a thing, as well as providing a clearer understanding of what is going on behind the scenes. So to that end, in this post I'll be describing the basics of the Markov process and why we would want to use it in practic...