Hello,
You should be able to migrate mixed volumes. The below steps were tested in a lab but i would recommend trying it ona test volume/qtree before proceeding with the full migration as for your data can be different.
1. Assuming that a volume with mixed security style is created on the destination system.
2. Scan the source and destination volume from XCP NFS client.
3. Copy the data using XCP NFS; once complete you can verify by checking the security style which should be showing UNIX
Example: file-directory show -vserver <vservername> -path <path>
4. Sync with XCP SMB with the -acl parameter; it will update the effective security style of the filer where it was NTFS along with the ACL info.
Thanks