Test tasks for product manager job applicants

product manager test taskThe test tasks aka product manager challenge or cases or homework for product managers job applicants became recently as common as the ones for designers or developers. While there’s nothing wrong with it as a practice these test tasks are often misused and candidates are taken advantage of.

Here’s why it’s a good idea to not proceed with doing test tasks. You will also find the real examples from Uber, Zendesk, Evernote and others at the end.

work for free

  1. It’s almost impossible to distinguish the real task from fishing for ideas when 99% of the tasks are specific real tasks related to the company business and the company has a pool of qualified candidates to take advantage as a free labor.
  2. The companies severely underestimate amount of efforts required to finish their task – 3-4 hours can simply mean 3-4 full days of work or even more. Half assed submission won’t get you anywhere and full-blown one is too much work (for free).
  3. As any homework test tasks are not a good indicator of a potential on the job performance – candidate basically has unlimited resources (friends, experts, nights, weekends) which is not often the case when working in the office and making decisions on the spot.

What to do instead?

    1. Pay candidates for their time
      That’s the fair approach if you want to keep using the test tasks in your hiring process. Hire a candidate for a day or two – they will do their best job and won’t feel at the end that they have been taken advantage of.
    2. Work with set of scenarios during the interview
      A couple of in-depth scenarios with follow up questions – better not directly related to your company – take Google, Facebook, Twitter, LinkedIn (considering you’re not hiring for these companies). On the spot thinking about the challenging situations proves to be even more valuable than the common test tasks.

Evernote Product Manager Challenge

This is to better understand how you would work through a standard problem that our PMs here at Evernote experience everyday. Select any Evernote product on any platform, but be specific. Write 1 paragraph describing a change you’d make to the product that would increase the first-week retention. Then write a brief outline of the process you’d go through from initial idea to release describing how you’d implement this change.

Babbel Product Manager assignment 

This exercise is designed to get a sense of how you approach product development and how you communicate your ideas to an audience. There is no right or wrong answer. Completing the assessment will help us to get an insight into your thought patterns on how to improve products.

To get started, please pick one digital product you are familiar with from one of these fields:

  • any kind of learning, so it does not necessarily have to be language learning (yes, you could also pick Babbel but it’s absolutely not required)
  • personal improvement, for example health or productivity

Task 1:
Suggest a feature to improve your chosen product. Describe the vision of the feature you want to develop and how you would measure its success.

Task 2:
Develop a user story map for your feature that includes a release roadmap and create a detailed user flow that covers the features of the first release.

Task 3:
Pick three user stories you would develop first from your user story map and write BDD styled detailed specifications (as if you would prepare the stories for grooming them with a development team).

Intercom Take Home Test

This is a short exercise that gives us an opportunity to see broadly how you think about Product, why things are built and how you think about defining problems. We believe this should take less than 3 hours to complete.

Part 1. A new Intercom competitor

Imagine you are starting your own company and are intrigued by the amount of activity in the market that Intercom competes in e.g. customer communication, marketing, support, product research.

As a starting point look at Intercom, Mixpanel, Urban Airship, Customer.io, Appboy, Sailthru, Reamaze, Kahuna, Helpscout, Helpshift, Olark, Zopim, Zendesk and any other new businesses in this space. Look for opportunities and gaps. Think about what problems these companies are trying to solve and how it could be done differently.

Please give us a paragraph or two on each of the questions below or feel free to communicate answers in a different format than text paragraphs:

  • What problem would you pick to solve for your new company?
  • What product would you design and build?
  • Assuming unlimited resources, what immediate team would you hire to get a first version launched?
  • What would your project plan look like at a high level? How many stages? How long would each stage take?
  • How would you measure whether that first version was a success?

Part 2. Software that excites you 

Looking at all the apps you use, which one or two do you think are incredibly well designed? What is it about them that makes them a step above all the other apps you use?

Uber Mobile App Health PM exercise

Our goal is to see how you think and attack problems, therefore I would recommend exhibiting the same level/quality that you would do in your actual job. Feel free to work in whichever medium(s) you are most comfortable – please ensure that everyone can review it and (most importantly) understand your thought process.

I’d like to emphasize that this should be a free standing product document so please clearly outline your assumptions + thought process. You won’t get the chance to present nor verbally explain it (sic!).

Take home tips:
Having seen a lot of these take ­home projects, I find that written docs, not PowerPoint, tends to lead to much more thorough exploration. The key is that your submission should be communicated so that anyone can review it and understand your thought process. Typically, the exercises are coming back ~3-4 pages in a doc format, please feel free to include wireframes, mock-ups, etc. The audience for this exercise are your colleagues from product & engineering.

I also recommend thinking deeply from a customer perspective. The rider perspective is likely more intuitive for the average person but I’d like to stress that our drivers are equally [if not more] important.

Our team mission is “our mobile apps are as reliable as running water” and we accomplish this by driving mobile app health via business & engineering metrics while remaining truly customer obsessed. We are the go-to team that brings visibility to metrics and provides fast actionable feedback to engineers, PMs, and operations across 400+ cities by focusing end-to-end on prevention, detection, mitigation, and resolution.

We work with a multitudes of signals (for example: crashes, network performance, app performance, etc) and scale of data points throughout the release cycle and in production. Our success will be determined by creating scalable platforms and experiences across multiple dimensions: SDKs, monitoring, alerting, ticketing, reporting, data-visualization, etc.

For this exercise, please focus on Crash Management as an area of opportunity for the team.

Roadmap & Metrics: How would you go about putting together a product roadmap  with focus on app stability? What is the feature set you’d execute on to meet your objectives and why? Uber’s primary metrics are trips, revenue, new riders. Show us how you can quantify the impact of the features you outlined.

Customer Centricity: Who are your customers? How will you prioritize amongst your customer requests?

Product Design: Pick one of your ideas, define the problem and resulting goals. Then propose a detailed solution. We’re looking to see how you communicate your ideas into an actionable format for a team to build.

Additional notes:

  • Feel free to pick a medium to share your thoughts: Google doc or Slide deck
  • Please don’t exceed 10 pages or 30 slides. Please be specific and deep with your recommendations.
  • Note that we are interested in your approach, passion and thought process in addition to the recommendations and conclusions.

Twitch writing sample for Product Manager

As a first step, I would like to request a writing sample. The writing sample prompt is as follows:

The drive to make more creators successful on our platform is continuous. However we have a complicated ecosystem of first and third parties offerings and a myriad of different kinds of creators. What is the right strategy for Twitch to win in this space? What measure of success would you suggest? Why should or shouldn’t Twitch pursue this?

Please write your narrative in a document that demonstrates your critical thinking and analysis. Typical lengths that we’ve seen are between 2-6 pages.

BusBud PM Challenge

You are the Sr PM of Web (Desktop & Mobile web) at Busbud. You know that 60%+ of our overall traffic comes via mobile web, so it is very important for the company.

A user is looking to buy a one-way bus ticket from Madrid to Barcelona on mobile web. After a Google search for “bus toronto new york”, he lands on this page.

Questions:

  1. Describe 2 key changes that you would make to the mobile web version of the route page above, or any subsequent screen in the purchase flow, if your only goal is to increase mobile app downloads for Busbud’s iOS and Android apps. (Note: The user must still easily be able to purchase a ticket using mobile web and your changes should not materially damage current mobile web conversion rates)
  2. Describe 2 key changes that you would make to the same flow if your only goal is to increase conversion rate in order to maximize ticket sales on mobile web.
  3. Assume now that both goals are of equal importance, and that you are looking to increase both mobile app downloads and mobile web conversion rate. Discuss what tradeoffs you would make between your proposed changes above, if any. (You can assume that travellers have varied preferences; some prefer apps while some prefer mobile web).

Zendesk Product Manager Design Exercise

Familiarize yourself with Zendesk’s multi-product administrative functions by creating a multi-product account. Create a new Support trial and then create Chat and Talk trials from within this Support account.

Evaluate in detail the current multi-product administrative experience and suggest improvements. Explain how you’d validate and prioritize these suggestions.

As you detail improvements to the multi-product experience, you may want to consider the following:

  1. How are new product trials are created and linked to existing products?
  2. What steps are part of the onboarding process?
  3. How do admins manage users across multiple products?

Deliverable

Provide a document detailing your analysis of the multi-product administrative experience. If you suggest UI improvements, low fidelity mockups are perfectly acceptable for this exercise – we are more interested in gaining clear insight into your proposed customer experience.

360dialog Product Manager Challenge

Answer the questions and feel free to include mock-ups, screenshots, or any other visual aids you feel are necessary. You may present your work in a format of your choice.

360dialog offers in-app advertising to its customers. One of the advertising products is an incentivized offer wall. Users get a list of offers that they can complete (e.g. fill out a survey or install an app) in exchange for a virtual reward in the app (e.g. 50 game coins). The user then gets acess to premium content without spending money.

  1. Assuming the offers shown to the user are currently in a completely random order, how would you improve the performance with a new ranking algorithm that sets this order in a more optimal fashion?
  2. How do you measure the results and how would you define success for the new algorithm (i.e. what kind of improvement would you expect)?
  3. Design an API that returns all the necessary information for the app developer to render the Offer Wall, so it can be adapted it to their own app design.

N26 Product Manager Case

We want to implement vouchers (e.g. Amazon) and special deals (e.g. supermarket) into our product in a way that it creates a sustainable value to our user as well as to our business to explore a new potential revenue stream.

Please create a few slides about your concept and approach how you would implement vouchers in a smart way into the N26 mobile app. Include a short project plan and major kpi’s you would consider. You can include wireframes, but it is not a must.

Thinkific assignment

For the assignment, please choose a favourite software product that you currently use, and write up a spec for a new feature that you’d like to see developed. Since we don’t have any existing user data, let’s just assume you are the only user (or maybe you do have that knowledge for the product you’ve chosen—in that case, you can definitely work off that if you want)!

Some examples of details to include:
Why this feature?
How does it work?
What are the launch plans?
How will you measure success?

…and whatever else you think is important! Length should remain under two pages, and it shouldn’t take you more than an hour to complete.

We’ve left it a little vague on purpose, so we can see your full range of creativity 🙂 We just ask that you state any assumptions you’ve made when building the spec so we can understand your thought process.

Yammer Product Manager Homework

Answer the questions and feel free to include mock-ups, screenshots, or any other visual aids you feel are necessary.

Instagram Stories

In August of 2016, Instagram released Stories.

1. What do you think Instagram’s hypothesis was for building the feature and what metrics do you think they used to measure results? How do you think they defined success before they shipped?

2. Based on your hypothesis and metrics from the first question, let’s assume the feature was not a success. What would that tell you? What would you do next?

3. Now, let’s assume Stories was a success. What are two or three ways Instagram could build on the success of this feature? For each feature idea you have, explain what your goals are, and describe the tradeoffs you’d evaluate before building that feature. If you could build only one, which one would it be and why? What would this feature look like? (Include rough mocks or sketches if possible).

Veeva Design Exercise

Part 1: Improve Document Templates
Document templates functionality is only minimally complete. It only consists of files, not metadata. There isn’t a easy way to tell templates are used a lot and which aren’t used at all. They are difficult to find and there isn’t an approval process within Vault. How would you improve this feature?

We do know the vast majority of users import files from their desktop into a particular document type. This would probably be the primary way it would be done even if the template functionality was complete, but templates are so limited that it is likely limiting adoption below its natural level.

Some of the feedback we have received includes:
Need to create and manage templates like regular documents (at least 8 customers)
Need approval process on templates (at least 4 customers)
Be able to track where-used for a template version – e.g. know which documents use which templates (at least 5 customers)
Plan on a ppt deck describing the key problem(s) to solve, key concepts introduced your solution, proposed screen changes (you can use simple drawing tools like what is in PowerPoint or Balsamiq or whatever your favorite tool is), and a sample user flow. Feel free to include other items as well related to the solution.

Part 2: Observations about Vault
Pick up to three things in Vault that you would improve to make Vault a better solution. Why those three? At a summary level, what would be your proposed solution. Can be simple things or complex things.

Pocket Gems Product Exercise

1. Ad Stuff (1 hr): These questions are to make sure you know what you’re getting into.
a. Play Episode, read a few chapters. Let’s say we’d like to show 2-3 ads per chapter immediately after a user spends a pass.
i. Propose a detailed design of what the user sees and when.
ii. What are some of the pros and cons of your designs from a user perspective? what about from a revenue experience?
b. What other Apps integrate video ads well? Why? (Please site references, share screenshots if necessary)
c. What do you think our eCPM is? Are we making our money by CPI or CPM?
d. What kind of advertiser do you think would want to advertise on Episode? What are their KPIs?
e. Page 45 of Mary Meeker’s 2016 Internet Trends report argues that there’s a large mobile advertising opportunity. What has been stopping advertisers from spending their money on mobile so far?
f. What excites you about what’s new (or coming soon) in ad tech? What should we do immediately?

2. Analytics (1 hr): For the following questions, please download the Excel file included in this link:  https://drive.google.com/open?id=0B6AaBVTikCFbUzhpSkhJY0J6eFU&authuser=0.  The Sessions tab contains all player sessions recorded for this game from 8/1/2014 to 11/1/2014.  The IAPs tab contains all in-app-purchase data for these players over this time period.
a. From these datasets, what are the key metrics you would want to look at when monitoring game health on an ongoing basis?
b. For each of these metrics, make a graph that shows how the metric has changed over the 3 month period included in the datasets.
c. Based on these graphs, what are some observations you have about how these metrics have fluctuated / changed?
d. What are some potential causes of the fluctuations / changes you identified in question (c)?
e. This link may be useful for your analysis: https://support.office.com/client/Create-a-PivotTable-to-analyze-worksheet-data-a9a84538-bfe9-40a9-a8e9-f99134456576

News360 Product Manager Test Project

Studies show that around 50% of medical patients don’t adhere to prescribed medication therapies – they forget to take medication, use the wrong dosage, etc. This obviously very negatively impacts the efficacy of their treatment. You’ve decided to tackle this problem by designing a platform to help doctors and patients manage medication prescriptions. The basic idea is:

An iPad app for doctors to issue prescriptions to their patients and track adherence
A smartphone app for patients that reminds them to take the proper medication on time, with correct dosage, and records compliance with these reminders
In version 1.0, you’re limiting the apps to the most common use cases, but with full functionality. Your go-to-market strategy is to market both direct to consumers, as well as to healthcare providers.

List the set of user stories required to describe the functionality required for the app
Create wireframes to show the main screens of the patient app
(any format is acceptable, feel free to make additional assumptions about the requirements, as long as you document them).

EA (Electronic Arts) Mobile Product Manager Quant Task

Question 1: Modeling
You are a Product Manager of a hit mobile free to play game responsible for projecting how a new feature will perform. Please use Excel to create a projection model for this feature with the following assumptions.
• Assume the feature will be active for one month (30 days).
• Assume the game is a city building title
• No need to include any costs.
• Use the following numbers:
Low Medium High
DARPU $0.10 $0.15 $0.20
Daily Spender Conversion 0.3% 0.7% 1.1%

The business model should include a Low, Medium, and High Case assumptions. You can use your best guess numbers. Feel free to expand the model to include any additional factors you deem relevant.
Please explain all assumptions you used.
Please create a PowerPoint presentation to answer questions 2-5.

Question 2: Live Game Operation – Data Analysis
You are the PM for a live game. Develop a report on the data sample attached below. Please provide an overview of trends and a method for tracking sales of individual items.
After analyzing the data, please provide a summary slide on findings in the presentation. Please send completed Data Sample excel along with the PowerPoint.

Helpful hints:
• Double click on the image below to open the excel sheet.
• Complete data in Raw Data tab as needed.
• Create a pivot table to help you analyze sales trends.
• Charts are wonderful

Question 3: Game Teardown
The candidate will be asked to create a PowerPoint presentation tearing down a top app store game showcasing their critical thinking, free-to-play chops, and design-focused thought process.
1. Provide a brief overview and evaluate key features of one of the titles below.
a. Clash of Clans
b. Gummy Drop
c. Tsum Tsum
2. How does this game acquire, engage, and monetize users?
3. How does this game formulate and reinforce long term play behaviors? Think about your experience playing the game and how these goals entice you to play longer in session, return to the game, and monetize. Provide a detailed example of what you feel is motivating players to drive deeper into the game and why.

4. FEATURE DESIGN: Provide a recommendation for a feature that you would implement to drive a specific KPI that you believe the game needs to improve.

5. GaaS DESIGN: Provide a recommendation for a live event you would create for the game to help drive engagement and monetization.
Discuss how the feature works and how it will drive engagement and monetization. Be prepared to discuss how would you increase the ROI on the time and resources required to develop the feature.

Question 4: Live Game Operations – Acquisition & Monetization
Your city builder game is struggling. Please address the following scenarios. Include 2-3 recommendations for each.
(A) You notice that installs have been steadily trending down in your game. What are some levers that you’d want to explore to increase installs? Examples should include both marketing and game development levers.
(B) You learn that once a user spends two or more times, she is significantly more likely to continue spending. The majority of users who spend, only spend once. What would you do to convert one time spenders to purchase their second time?

Question 5: Pricing
You are the PM for a city building game. You need to come up with a pricing for your virtual currency (coins). The current currency packs available in your game are listed below.
Is this is a good pricing scheme? If applicable, how would you improve it?
Item Small Pack Medium Pack Large Pack
Price $1 $2 $5
Quantity 10 coins 20 coins 50 coins