For Jupyteach development I have been studying the excellent Livebook project's source code. As I am somewhat new to Elixir development, I have been learning a great deal from the pros. Here's what I learned today...

Linked Lists

In Elixir a list ([a, b, c]) is implemented as a linked list. One implication of the linked-list structure is that prepend operations are O(1), but append or insert operations are O(n).

To encourage developers to work efficiently with this data structure, Elixir's pattern matching supports some special syntax exists that makes it easy to take the head of a list (first element) and the rest of the list like so [x | rest]. In this example x will be the first item of the list and rest will be a list with everything else.

It is also very convenient to prepend a single element to the front of a list by doing new_list = [x | old_list], which will be an efficient O(1) operation.

Livebook Notebooks

In livebook's source code a notebook struct organizes sections and cells as follows: notebook = %{sections: [%{cells: []}]}.

That is, a notebook has a list of sections, each of which has a list of cells.

In livebook, you are allowed to move cells up and down by a given offset. This offset might cause the cell to cross section boundaries. The function move_cell/3 is responsible for handling this operation. Here's the code:

def move_cell(notebook, cell_id, offset) do
  # We firstly create a flat list of cells interspersed with `:separator`
  # at section boundaries. Then we move the given cell by the given offset.
  # Finally we split the flat list back into cell lists
  # and put them in the corresponding sections.

  separated_cells =
    notebook.sections
    |> Enum.map_intersperse(:separator, & &1.cells)
    |> List.flatten()

  idx =
    Enum.find_index(separated_cells, fn
      :separator -> false
      cell -> cell.id == cell_id
    end)

  new_idx = (idx + offset) |> clamp_index(separated_cells)

  {cell, separated_cells} = List.pop_at(separated_cells, idx)
  separated_cells = List.insert_at(separated_cells, new_idx, cell)

  cell_groups = group_cells(separated_cells)

  sections =
    notebook.sections
    |> Enum.zip(cell_groups)
    |> Enum.map(fn {section, cells} -> %{section | cells: cells} end)

  %{notebook | sections: sections}
end

Comments:

• separated_cells is a flat list of cells, where in between each section's cells is the atom :separator. It looks like [cell1, cell2, :separator, cell3, cell4, cell5, ..] where cells 1 and 2 are in the first section and cells 3 to 5 are in the second section.
• idx is the original index of the cell we want to move (found using cell_id)
• new_idx is the new index where we would like the cell to end
• List.pop_at and List.insert_at are used to remove the cell from idx and put it back in at new_idx
• cell_groups will return list(list(Cell.t())), for the cells that belong to each section
• Finally the last two chunks of code update the list of cells in each section and then the list of sections in the notebook

Regrouping cells

The part I want to focus on is group_cells/1. The implementation is a master class in pattern matching and efficient list operations in Elixir. Here's the code

defp group_cells(separated_cells) do
	separated_cells
	|> Enum.reverse()
	|> do_group_cells([])
end

defp do_group_cells([], groups), do: groups

defp do_group_cells([:separator | separated_cells], []) do
	do_group_cells(separated_cells, [[], []])
end

defp do_group_cells([:separator | separated_cells], groups) do
	do_group_cells(separated_cells, [[] | groups])
end

defp do_group_cells([cell | separated_cells], []) do
	do_group_cells(separated_cells, [[cell]])
end

defp do_group_cells([cell | separated_cells], [group | groups]) do
	do_group_cells(separated_cells, [[cell | group] | groups])
end

We enter with group_cells and pass in separated_cells, which is the re-ordered list(Cell.t() | :separator).

The code then reverses that list and begins a series of recursive calls into the do_group_cells/2 function. In all clauses of this function the first argument is a flat list of cells and the second argument is a list of lists of cells. The outer list is for sections. Each inner list is a group of cells in that section.

We begin our time in do_group_cells/2 by passing the reversed separated_cells list as the first argument and an empty list ([]) as the second.

Note: I originally started this deep dive by trying to figure out why the list of cells was being reversed and how it was ever "un-reversed".

Recursion and Pattern Matching

This will match either the 2nd or 4th clause (due to the empty list as the second argument).

If our reversed list happened to begin with a :separator, we go to the 2nd clause. Here we throw away the separator (by keeping only the tail of the first argument) and then make a recursive call updating the second argument to be [[], []].

Why [[], []]? To see we need to follow the recursive call into the 5th clause (why 5th and not 3rd? because we won't have two :separator atoms in a row).

The 5th clause is the work horse. It uses pattern matching to split the first argument into head (called cell) and tail (called separated_cells) and splits the second argument into head group (list of cells in current section) and tail groups (the other sections and their cells). The only line in the body of this function is to make another recursive call to do_group_cells/2 passing separated_cells as the first argument and [[cell | group] | groups] as the second. This second argument does two things: (1) puts the single cell at the head of the current group and (2) puts this updated group ahead of existing groups.

Comments:

• By always peeling cell of the head of the list of cells and then adding it to the front of the current group (updated group to the front of the list of groups), we end up undoing the List.reverse from group_cells/1
• By always passing separated_cells as the first argument in the recursive call that list shrinks by one each time until it is empty. When it is empty we fall into the first clause and recursion ends.

To get here we followed the 2nd clause. Suppose, on the other hand that we end up in the 4th clause to begin the recursion. This happens whenever the reversed list of separated cells does not begin with :separator. In this clause we match the head of our cells to cell and the tail to separated_cells. We then do do a recursive call into do_group_cells/2 separated_cells as the first argument and the second argument as [[cell]]. This will match the 5th (or 3rd if the new head of separated_cells is :separator) clause and we go to work as described above!

The final piece of the puzzle is the 3rd clause. This matches when the current list of reversed cells begins with :separator. Here the recursive call will throw away :separator, pass the tail of the cell list in the first position and then [[] | groups] as the second. This sets up a new empty cell list for the current group and we recurse back into the 5th clause.

Learning from the best

Functional programming is a powerful approach to modeling and manipulating data. Elixir's implementation allows for concise and expressive code, but it can seem foreign to someone like me with a stronger background in languages like Python, Julia, Javascript, and Go.

Thankfully the team behind livebook are experts and have made their code available and open source. It has been exhilarating to study their code and feel like I'm taking a class from the best in class. Open source for the win!

Postscript

I made use of the Cody tool from Sourcegraph while I was studying the livebook source code. While not not perfect, Cody was an excellent study buddy as I dug into this new library.

Data, like crude oil, holds vast potential but requires refinement for value. Its worth is in the insights gained from analysis, influencing strategic and operational choices. Furthermore, with the integration of machine learning and artificial intelligence, data can be leveraged to predict trends, optimize processes, and make smarter decisions. Data-driven organizations, bolstered by these technologies, are more successful, innovative, and prepared for the future. However, before harnessing this potential, the challenge of data storage and access must be addressed, which is the primary focus of this blog post.

In today's interconnected world, modern companies face the intricate challenge of integrating seamlessly with a myriad of external applications, including marketing tools like Mailchimp, payment providers like Stripe, and CRM systems like Salesforce. Traditional business infrastructures often falter when interfacing with these diverse platforms, lacking in efficiency and cost-effectiveness. Recognizing this gap, the team at Valorum Data specializes in advanced integration strategies. Our latest endeavors are geared towards crafting bespoke solutions tailored to specific organizations and projects, ensuring they can smoothly interact with these vital external systems and achieve transformative outcomes.

Building Customized Data Warehouses

When it comes to assembling the perfect data warehouse, one size does not fit all. In the past few decades, we as an industry have learned a few strategies for effectively managing and maintaining a data warehouse. On such pattern is the Extract, Load, Transform (ELT) strategy – a variant of the traditional Extract, Transform, Load (ETL) process . At Valorum, we create ELT pipelines that extract data from a variety of sources/APIs/databases/external applications, load all data into a data warehouse, and then transform the data within the warehouse. This powerful paradigm is deployed at many of the leading data-first firms today. However, the majority of companies do not operate at a large enough scale to justify having an in-house data engineering team. This is why Valorum Data has been building custom data warehouses for clients, tailoring each one to suit the unique needs of the organization.

A crucial part of any data warehouse is the choice of where to store the data. We utilize different systems such as TimescaleDB, Snowflake, and BigQuery, depending upon the volume, variety, and velocity of the data to be managed. These technologies have been designed with scalability at their core, making them instrumental in managing complex business data and supporting advanced analytics.

Streamlining Data Sync and Transformation

After the data warehouse is set, seamless data integration is paramount. To achieve this, we use Airbyte, an open-source data integration platform, to sync data from APIs, customer databases and other systems. Furthermore, Valorum has leveraged its expertise to write custom Airbyte connectors for improved accessibility and smoother data integration.

Data transformations are crucial in a data warehouse to enable effective decision support. For this, we use dbt (data build tool), which allows us to perform transformations within the warehouse post extraction and loading. The power of dbt lies in its ability to manage complex transformations and enable version control, all while maintaining transparency through automatic documentation. By converting raw data into a more suitable form for reporting and analytics, dbt takes data from accessible to actionable.

Creating Custom Applications and Reports

The end goal of all these processes is to generate actionable insights. To that end, we build custom applications and automated reports on top of the data warehouse. Leveraging a variety of Python tools, we can present the data in the most effective and easily digestible format: as dashboards, interactive visualizations, automated reports, and more. These tools give businesses the power to actively make use of their data in real-time, making them more dynamic and responsive.

With the clear, accessible data at our disposal, we work with businesses to utilize an array of machine learning and artificial intelligence algorithms to delve deeper into the data. Whether you want to predict future trends, identify anomalies, or segment your customer base, our data infrastructure creates the optimum environment for these predictive analytics. Machine learning and AI thrive on big, clean data, and that's precisely what we provide, thus allowing organizations to make the leap from data-driven to AI-driven. This ushers in the next level of intelligent decision-making, proactively leveraging data to gain a competitive edge.

Join the Fun

In conclusion, every organization's data is a treasure trove of insights waiting to be unearthed. However, unlocking the full potential of this data can be challenging without the right strategy and tools. With an organized approach to data storage and access, Valorum Data is helping organizations build the necessary infrastructure to become data-driven.

As we continue expanding and refining our strategies, we invite you to be part of our journey. If you're interested in learning more about our work or discussing how we can support your organization's data needs, please reach out to the Valorum team for a consulting engagement. Whether you are just starting a data strategy or well on your way, we can help you reach your goals. Alternatively, subscribe to our newsletter to stay up-to-date with our latest advancements, and have useful data insights delivered straight to your mailbox.

At Valorum Data, we don't just provide services, we equip you with the power to transform your data into a potent strategic advantage. Regardless of the complexity of your data landscape, we can help you navigate through it, unlocking significant value along the way. Harness the power of your data with Valorum Data, and make your organization truly data-driven.

The Valorum team has a passion for learning and teaching. We love to explore new ideas and help others feel the energy that comes from gaining knowledge. Over the past 10 years we have continuously taught courses at a number of universities around the world. Our courses are mostly Python based (some Julia) and cover topics at the frontier of economics, machine learning, and data science. During our experience, we have searched for the best way to teach in person and remote courses on these topics.

We've tried everything.

We mean everything...

• traditional learning management systems (LMS) including Canvas, Moodle, Google Classroom, and Brightspace
• developer tools like GitHub and GitHub Classrooms
• data science specific platforms like Cocalc and Jupyterhub.

Each of these tools is excellent in many ways, but none of them was quite right for the way we like to teach courses. We build all our lecture notes in Jupyter notebooks. We like to mix explanation -> example -> exercises/practice freely in the notes themselves. We encourage students to alter the codes and explore different options and possibilities. We often flip the classroom, so great video support is a must.

So, we decided to create Jupyteach. A new LMS, streamlined for our needs. It doesn't do most of the things some of these other platforms do. But what it misses in terms of features it makes up for in ease of use and powerful interactivity.

Jupyteach is under active development (and use at multiple universities including New York University and Peking University HSBC Business School) and is in the alpha stage. In this post we will give an outline of our current progress as well as hint at future plans and ideas. We will provide updates in future posts.

Features

We are excited about Jupyteach, primarily because it does exactly what we need it to. No more, no less. The key features already implemented in Jupyteach include the following:

• User management: users can sign up via email+password or via email and passwordless codes sent to their inbox
• Course management: users that are instructors can create courses. Within a course instructors can invite students other instructors and TAs, upload a syllabus, and manage course content (see below)
• Communication:
  • Announcements: instructors can create announcements that appear on the announcements page of their course and are also delivered to all students via email
  • Discussions: all course participants can start a discussion thread
• Content: content on Jupyteach is organized in a two-level hierarchy. The smallest unit of content is a content block. Content blocks can be organized into lectures. Content blocks can be one of a few different types
  • Notebook: a Jupyter notebook that will be rendered in a live computational environment powered entirely by the user's browser using the JupyterLite project. This means (1) no installation step for students (2) no server management for instructors and (3) interactive lecture notes!
  • Video: an embedded video hosted either on YouTube or Vimeo
  • Markdown: a standalone markdown document that is rendered on its own page
  • Assignment upload: A place for students to attach/upload Jupyter notebooks they have completed as homework
  • Link: a link to an external resource such as a data file or pdf
  • Quiz: an interactive, auto-graded quiz (see below)
• Quizzes: automatic grading is a must for our teaching. We like to give students many opportunities to practice the concepts they are taught, but doing so typically comes at the cost of more time spent grading assignments. So, Jupyteach has quiz functionality. Quizzes are composed of multiple questions, all of which are automatically graded as the student submit responses. Questions can be of multiple types:
  • Single selection: a user is presented with a prompt and a list of options. They must select the single correct option
  • Multiple selection: a user is given a prompt and a list of options. They must select all correct options
  • Code: the user is given a description of a programming task and some starter code. They must modify the code so it correctly solves the task. Correctness is verified by running the user's against a test suite set up by the instructor. The Python test code is run in the student's browser, making the automated quiz system cheap, efficient, and scalable.
  • Fill in the blank: the user is given a problem description and a piece of code that has missing values. They must fill in the missing values. Correctness is verified using a test suite using the filled-in code similar to the Code questions

Real Time

One unique aspect of Jupyteach is the technology we chose to build on. We are using the Phoenix web framework – a powerful web framework for the Elixir programming language. Elixir is built on top of Erlang/OTP and the associated BEAM virtual machine. The BEAM is known for being extremely resilient and scalable. What this means for Jupyteach is that we will be able to build real time features (think google docs) with shockingly little effort.

We have big ideas and plans for this, but none of them are quite ready to talk about yet. As soon as we finalize the core feature set (9 of 11 items done), we plan to shift engineering efforts to real time system. Keep an eye on this space!

Looking Forward

We have many excited things planned for Jupyteach and can't wait to share them with you. We'll be posting here regularly, so subscribe today to stay up to date!

Welcome

Greetings to the curious, the learners, and the experts! We're thrilled to have you join us on this exciting journey of exploration into the ever-evolving world of data. If you have an insatiable appetite for the technical intricacies of data science, machine learning, data engineering, programming, economics, and statistics, you've landed in the right place.

Who Are We?

Established in 2018, Valorum Data has carved a niche in the data consulting arena. With hands-on experience in a diverse range of industries—including healthcare, power and energy, retail, finance, education, and manufacturing—we've developed a solid understanding of how to unlock the latent power of data across disciplines. Our core services reflect our dual commitment:

1. Training and Coaching: We believe in empowering individuals and teams, equipping them with the skills and knowledge to navigate the intricate landscape of data.
2. Custom Solution Design, Implementation, and Support: Our expertise isn’t limited to guidance. We roll up our sleeves to design, execute, and maintain bespoke data solutions tailored to specific challenges and industry nuances.

At Valorum, it's not just about the depth of our knowledge; it's about the breadth of our experience and the practical solutions we bring to the table.

What's in store for you?

This blog promises a blend of the technical and the theoretical:

• Code Away: Dive deep into rich content full of code snippets and practical examples, tailored to help you grasp complex concepts.
• Theory Meets Practice: As we delve into advanced topics, we'll ensure a balanced marriage between underlying theories and their real-world applications.
• Interdisciplinary Insights: The fusion of programming with economics, statistics with machine learning, and so much more, awaits you. Our posts will often crisscross these domains, reflecting the interdisciplinary nature of today's data challenges.

Why should you join our journey?

Because we're passionate, not just about crunching numbers, but about understanding them, analyzing them, and sharing the knowledge we garner. Every blog post is an opportunity for us to learn alongside you, to question, to discover, and to grow.

And while the realm of data might sometimes feel overwhelming, remember: every expert was once a beginner. Whether you're just starting out or have been in the field for years, there's always something new waiting on the horizon.

In the bustling world of AI, there's more than just algorithms and code. It's about unlocking new avenues for businesses, catalyzing growth, and driving innovation. With us, you'll not only learn the technicalities of AI but also explore its myriad business applications. From streamlining operations to gaining a competitive edge, delve into the transformative power of AI in reshaping industries and redefining success.

Engage with Us!

Dive into the conversation and become an integral part of our community. We value every insight, feedback, and perspective you bring to the table, and we're eager to hear from you. Whether it's a topic you'd like us to cover, a comment on a recent post, or a challenging question you've encountered, share your thoughts.

The world of data is ever-evolving, and we want to ensure you're always in the loop. By subscribing to our newsletter, you get a curated selection of our latest insights, tutorials, and discussions delivered right to your inbox. Stay connected, stay informed.

Beyond this blog, our proficiency spans a wide spectrum of services. Whether it's teaching advanced courses, providing one-on-one or team-based coaching, or crafting bespoke development solution; we have the expertise to support your needs. For businesses and individuals alike, our team is geared up to offer tailored assistance, ensuring that you harness the full potential of data. Reach out, and let's discuss how our diverse offerings can align with your objectives and propel your projects forward.

To infinite learning and beyond!

Warm regards

Valorum Data