Why doesn't concatLayer in Deep Learning Toolbox concatenate the 'T' dimension?
1 view (last 30 days)
Show older comments
Hello,
While implementing a ViT transformer in Matlab, I found at that the concatLayer does not concatenate over the T dimension. This is needed to concatenate the class token with patch tokens, since the natural representation is CBT with C corresponding to features, B to batch and T to token within a batch (this is also the canonical representation in the attention function).
It's possible to work around this by hacking to e.g. SCB, but then other problems pop up which also need to be hacked around.
Thx
0 Comments
Accepted Answer
Ben
on 14 Mar 2023
You can create a layer that concatenates on the T dimension with functionLayer
sequenceCatLayer = functionLayer(@(x,y) cat(3,x,y));
This will work in dlnetwork to concatenate two CBT dlarray-s.
Since you're concatenating the class token, it might also be worth considering creating a custom layer that has the class token embedding as a Learnable property, and performs the concatenation in the predict method.
3 Comments
Catalytic
on 23 Mar 2023
Edited: Catalytic
on 23 Mar 2023
@John Smith - Since Ben's answer yielded a solution for you, you should hit the Accept this Answer button, and likewise with other answers you might not have accepted.
Artem Lensky
on 19 Aug 2023
Are there any plans to make concatenationLayer support concatetnation along the T dimension?
More Answers (0)
See Also
Categories
Find more on Image Data Workflows in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!