The Multiple Spanning Tree topology is contained with the internal multiple spanning tree area. The region boundary is defined when a port connects to a switch that is in a different Multiple Spanning Tree region or a non-MSTP port (Such as 802.1D or 802.1W)
The instances will never interact outside of their region. MST enabled switches have the ability to detect Per-VLAN Spanning Tree (PVST) switches at their boundaries, propagating their common spanning tree configuration sourced from their internal spanning tree topology towards the PVST topology. This feature is known as the PVST simulation mechanism.
In the other direction with BPDUs received into a Multiple Spanning Tree configuration from a PVST topology, the MST configuration maps the BPDU from VLAN1 into the internal spanning tree instance.
There a couple of design considerations to take note of when it comes to mixing spanning tree protocol modes:
If the MST Region is the root bridge across the topology
The MST enabled switch being the root bridge allows all region boundary ports to flood the same BPDU to all VLANs in the PVST topology.
If the MST Region is not the root bridge across any VLAN
The MST enabled switch can only block or forward for all VLANs. The VLAN1 BPDU from per-vlan spanning tree is used for the internal spanning tree process in multiple spanning tree.
This means the VLAN1 topology is translated to every VLAN in the multiple spanning tree protocol.
If the MST switch detects a more preferred BPDU on a region boundary port, the switch will block this port and be placed into a root inconsistent state. This to help keep a loop-free topology. This is known as a PVST simulation check