Work

Hi, I'm Boris. I am an Engineer turned Product Manager. I believe that great products are built through deep collaboration and a good methodology.

Below are a few case studies of products I have owned or had a significant hand in developing.

E-Commerce Search @ Educents

 
search bar - v2.png
 

At Educents, I designed and co-implemented our search algorithm. As we grew from e-commerce to marketplace and added thousands of products to our site, the need for a refined search experience grew.

Educents is the first marketplace for parents & homeschoolers to find the best educational products. I worked with them from pre-seed to post series A, alongside a team of engineers and data scientists. The search project described here spanned 2 years of iteration.

Our search ultimately grew to account for 15% of sessions and over 40% of revenue on the site. Looking back on it, I think we nailed the balance of iterative process + continual refinement with search and grew a successful product. Here’s how we did it.


Context

Goal: The goal of search is for the user to find something they want to buy. 

Use Cases:

  1. Find a specific product
  2. Find a specific seller
  3. Find a product that fulfills your needs/intent without a specific product in mind (browsing).

KPIs: add to cart conversion %

Corollary KPIsAOV, Margin (KPIs that we watch to make sure they don't plummet, while aiming towards our KPI).

Approach: 

  • Leverage existing open-source technologies for the fastest initial release
  • Understand our demographic's unique needs, and reflect those via algorithm refinements
  • Build a flexible system, optimizing for speed of iteration and learning
  • Add complexity over time, and get started quickly

The Process

v1 - "Ship It"

"The worst product is the one that doesn't exist." The very first version of our search, back in early 2015, was a simple MySQL %like% query that connects to the database. 

It was simple and quick to implement. More importantly, it allowed us to start learning from day one, while we developed something better.


v2 - Make it Better

The second version we launched was powered by Elasticsearch, one of the leading open-source technologies available for search.

We didn't start off utilizing all of Elasticsearch's capabilities, but we knew that getting on this platform would enable us to build on our algorithm for many years to come.

This turned out to be a great decision, and ES (Elasticsearch) allowed us to save many dev. cycles and build out highly complex algorithms as we scaled.


v3 - Get Feedback & Iterate

This is when we really got our hands dirty. We did internal testing. We got feedback from our marketing, operations, content, and engineering teams. We opened up a Facebook group for our sellers and power-customers to give us their thoughts. We tested against different user flows - finding products, sellers, and browsing.

Then we started tweaking our algorithm. Elasticsearch gives you tremendous capabilities if you understand relevance, indexers, tokenizers, and queries.

We built out weights and filters based on product attributes and customer interest (e.g. reviews). We designed for different cases: making sure the user could find a specific product, storefront, or browse and find the highest-quality products on our site.

Slowly, we started to see improvements and our usage grew from a measly 1% to 6% and more. People were starting to come back and use search.

Searching for "times tales" - a product and seller - shows those results at the top

Searching for "times tales" - a product and seller - shows those results at the top


v4 - A/B Testing

Soon, our search usage grew enough that we could do quick, iterative, statistically-significant A/B tests. We embedded the concept of A/B testing and algorithm design into our core search code and got busy testing: first the UI, then refinements to our algorithm. That's how we landed on our new left-hand rail design.

A/B testing allowed us to experiment with more advanced algorithm design as well. We tested impressions, conversion rates, product placement, even advertisements. We learned a lot and our algorithm conversion grew leaps and bounds - as we saw 10%, then 20% and 30% relative increases in our add to cart conversion %. Our customers were finding what they needed faster, and our search was humming.


v5 - Taxonomy

A robust taxonomy underpins any great marketplace. We'd pushed it off as long as possible to collect data about what our customers were searching and browsing for, since taxonomy can be a big time investment.

With lots of raw data as well as team, customer, and seller input, we were able to build a comprehensive taxonomy. This would go on to tie our site together - our content, community, sellers, reviews, search and discovery - and form the foundation of our strategy moving forward.

It also empowered our customers to search for exactly what they needed, faster. Select the grade, pick the skills or subjects you are interested in, filter by product type, and you've got what you need.

Besides laying a long-term foundation for our business, our taxonomy also enabled a richer left-nav UI experience that lifted our conversion rate a further 20%.

 
taxonomy-1.png

 
Anything you can imagine is real
— Pablo Picasso
 

Future - TBD

Every product should be about iterating & improving. As such, our work is never really done.

What's the future of our search? There are many options. If I were to suggest one candidate for improvement, it would be building a system for continuous and automated algorithm testing. To have the algorithm self-learn what works, what doesn't, and re-balance its own weights over time - testing itself against shifting consumer demands and an ever-growing catalog.


Results

In the end, we had a search algorithm that powered 40% of our revenue, 15% of sessions, and that saw conversion rates of 20-30%. Of course, it's a success that depended on having great products underneath it. It would have been impossible without the hard work of our engineers, data scientists, and account managers who worked tirelessly to get incredible products & sellers onto our site. As most things in product design, search is just one part of a larger, thriving ecosystem - but a key component nonetheless.