We turned MarTech into a chatbot. This is what we’ve learned (so far)

OpenAI’s release of ChatGPT got everyone in the tech world thinking, ourselves included, about its impact on everything. We started to wonder, and ask some big questions about how ChatGPT and technologies like it would shape the future of marketing, websites and customer experience. We asked:

• Will this type of generative AI replace websites one day?
• Will this technology have the capability to be useful, or helpful for marketers?
• What are some of the emerging use cases we should explore?

There were so many more questions that I even created an internal list of use cases, brainstorming out loud how we might use these technologies to help us run Third Door Media, to produce better content, and ultimately help you, the MarTech community embrace not only the change that generative AI represents but also the opportunities.

The best way to do that was to get our hands metaphorically dirty and build our own bot. So John Blanchard, our senior director of marketing technology, and I did. We built an index of all the MarTech.org pages and posts and embedded them into MarTechBot which sits on top of ChatGPT. 

It’s only been a few days but here are some of our initial learnings:

Humans like to break things

We humans naturally try to break things (perhaps in order to confirm our biases?). Many users sent us comments that because MarTechBot wasn’t perfect or always correct, it must not be very good. It’s true that the results can be shaky depending on what you type into the box. But it’s also true there have been hundreds of conversations with MarTechBot, some of which have been multi-question threads with increasing specificity. For example in a single chat, a user asked these two questions back to back, getting answers for both:

• What are the key considerations before we conduct a full marketing technology audit?
• What are the key questions to ask the internal stakeholders?

Speaking of which, we discovered a bug that throws off a “token error.” We know why it’s happening but can’t fix it quite yet. Stay tuned! 

No one seems to know how any of this actually works

This is new tech. No one seems to have a full mental model of what these new tools are capable of, or how exactly they work. 

It’s not a search engine, despite feeling like one.

I heard Baratunde Thurston on the Pivot podcast say that it can feel like you are casting spells when using these tools. He’s right. Generative AI tools are quickly becoming like a “mech” unit, arming me with capabilities and power I am still uncovering and learning to use.

But those powers come with some real caveats. One is that we don’t exactly understand why the language models and bots answer the way they do. With great power comes great responsibility (and a requirement to read closely and check your work!).

MMM, what?

Here’s a real example. A user asked MarTechBot to “Tell me what I need to know about MMM?” and got back the following:

AI: Marketing Mix Modeling (MMM) is a technique used by marketers to measure the impact of advertising and promotions across channels while controlling for external factors outside of a brand’s control, such as inflation or consumer sentiment. The goal of MMM is to help marketing leaders plan future marketing spend and measure the performance of past investments. MMM can be used in three ways: as a scorekeeper, to show the overall incremental impact marketing investments are having on the overall business; as a forecaster, to predict the outcome that raising or lowering marketing budgets will have on marketing’s contribution to the overall budget; and as a coach, to suggest shifts to current marketing investments that improve performance. MMM is a core measurement capability that offers significant opportunity to increase returns on media investments, but the models require consistent reevaluation to consistently deliver the expected benefits.

That same query in ChatGPT got:

MMM can refer to different things, depending on the context. Here are three possibilities:

MMM as a financial abbreviation:

MMM is the stock ticker symbol for 3M Company, a multinational conglomerate corporation that produces a wide range of products, including adhesive tape, medical products, and office supplies. The company is listed on the New York Stock Exchange (NYSE).

MMM as a cryptocurrency:

MMM Global is a self-described “community of people providing each other financial help on the principle of gratuitousness, reciprocity, and benevolence.” It was founded in 2011 by Sergei Mavrodi, a Russian businessman, and purportedly operates as a decentralized peer-to-peer network. MMM Global uses its own cryptocurrency, called MMM Coin, as a means of exchange within the community.

It’s worth noting that MMM Global has been associated with multiple pyramid schemes and Ponzi schemes over the years, and has faced legal action in several countries.

MMM as a texting abbreviation:

In texting and online communication, MMM can be used as an abbreviation for “mmm-hmm,” which is a way of indicating agreement or acknowledgment. It can also be used to convey a sense of appreciation or enjoyment, as in “MMM, this pizza is delicious!”

Bard gave me this result:

MMM stands for 3M Company. It is an American multinational conglomerate corporation headquartered in Maplewood, Minnesota. The company’s diversified technology portfolio includes industrial, safety, and consumer products. 3M has over 90,000 employees and operates in more than 70 countries.

And a bunch of other info about 3M. You get the picture.

Does that mean MarTechBot is effectively “pre-prompted” because we trained it on MarTech.org content? I think so. The implications that context matters and prompting makes a difference are wild to contemplate. It makes logical sense that MarTechBot should answer this way, but then again, there was no prompt, just the letters MMM. Wild.

People aren’t sure how to use generative ChatBots, yet

Not only do we not really understand how these things work, people aren’t sure what to do with them either. 

The optimal interactions seem to be multi-layered learning experiences. Typing “SEO strategy” is simple, but not nuanced. More value is extracted instead by being more specific. “Give me an SEO strategy for a small flower shop located in Portland. We have a small influencer program and blog regularly” which will likely yield more useful results. Asking follow-up questions can help refine and fine-tune results. Again, it’s not a search engine. It’s something different. Alien almost. 

The more I experiment, the deeper the rabbit hole. Yesterday, I used a prompt to create a quiz for myself, testing my SEO knowledge. Today, I used a prompt to practice my selling skills by simulating a conversation with a CMO of a national brand who I was trying to convince to beta test my fictional analytics software platform.

The implications of this are incredible. Bill Gates might be onto something here when he talks about AI replacing teachers.

As we humans learn to use (and live with) these tools, new use cases will emerge, and emerge quickly. Earlier I wrote about the idea I had about creating a self-testing quiz and simulating a sales pitch. Today, I used Google Bard to create a simple Javascript calculator to help me think about different email scenarios and then created an interactive podcast start-up checklist. 

The possibilities are endless. 

Where to from here?

To be frank, one of the reasons we built MarTechBot was to bring awareness to MarTech.org. Instead, it’s become a collaboration with you, the MarTech community. Feedback has been mixed. Some are grateful we launched MarTechBot, and others are frustrated with the token bug or how slow, in comparison to ChatGPT, the bot is. Each comment is valuable from each we learn and grow. The joys of beta launching something, am I right? 

We’re looking at ways to improve the bot and want your feedback. Give MarTechBot a shot, and find the feedback form on that same page. I look forward to hearing from you.

I’m posting about the journey on LinkedIn, you can follow me there for more real-time info and chat.