You may limit the files that user can upload based on filename. To enable these settings go to "Banned Files" node for desired domain. If enabled then uploading files matching any of the regular expressions will be disallowed.
For example, in Figure 132 below, users may not upload files with a .txt extension.
Figure 132
See also