This post explains some of the differences between Quest Machine and the Dialogue System for Unity.
Quests
Quest Machine aims to make quest design easier, with a node-based quest editor with live runtime view, and also let you procedurally generate new quests at runtime. Another strength is the efficient and customizable way that it handles the gameplay part of quest progression, which requires no scripting unless you want to create your own sub-asset types.
The Dialogue System's runtime quest control components are simpler and less extensive. While it's possible to create runtime quests in the Dialogue System, it's not built in. You have to use scripting or visual scripting.
Dialogue
Out of the box, Quest Machine's dialogue works like many MMORPGs, with "Accept" and "Decline" buttons rather than interactive branching conversations. One way to add branching conversations is to use the Dialogue System integration.
The Dialogue System is all about branching conversations. It also supports barks (one-liners).
Customizable UIs
Both support customizable UIs.
Saved Games
Both can save and restore their data.
Localization
Both have built-in support for language localization.
Third Party Support
Both are designed for easy extensibility and integration with other assets. The Dialogue System currently has the larger number of third party integrations, but Quest Machine's count is growing, and it already integrates with most of the big hitters such as PlayMaker and ORK Framework.
The Dialogue System can also import content from external applications such as articy:draft, Inklewriter, and Chat Mapper, which Quest Machine doesn't support yet.